Nov 02 21:08:07.461520 ip-10-0-137-1 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:08:07.889958 ip-10-0-137-1 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:07.889958 ip-10-0-137-1 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:08:07.889958 ip-10-0-137-1 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:07.889958 ip-10-0-137-1 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:07.894055 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.893984 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.894055 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894055 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894058 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894062 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894066 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894069 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894072 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894074 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894077 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894080 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894082 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894085 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894088 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894090 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894093 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894095 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894105 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894108 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894111 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894114 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894116 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.894116 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894119 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894122 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894125 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894128 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894130 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894133 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894135 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894138 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894142 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894146 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894149 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894151 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894154 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894157 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894159 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894162 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894164 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894167 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894169 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.894582 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894172 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894175 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894179 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894182 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894185 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894188 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894191 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894193 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894196 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894199 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894201 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894204 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894206 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894208 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894212 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894215 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894217 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894220 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894222 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.895009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894225 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894227 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894229 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894232 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894234 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894249 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894252 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894254 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894257 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894260 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894262 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894265 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894267 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894270 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894273 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894275 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894278 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894281 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894283 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.895458 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894285 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894288 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894291 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894293 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894296 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894298 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894301 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894303 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894307 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894310 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894313 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894315 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894318 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894320 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894323 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894325 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894328 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894330 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894333 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894335 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.895893 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894339 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894341 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894344 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894346 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894349 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894351 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894354 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.894356 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895092 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895098 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895101 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895104 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895107 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895109 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895112 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895115 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895117 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895120 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895123 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895126 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.896380 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895129 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895133 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895136 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895138 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895143 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895146 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895149 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895151 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895154 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895156 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895159 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895161 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895164 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895166 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895169 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895171 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895174 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895177 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895179 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.896876 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895182 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895184 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895187 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895189 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895191 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895194 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895196 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895199 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895201 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895204 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895206 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895209 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895212 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895215 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895217 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895220 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895222 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895225 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895227 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895230 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.897351 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895232 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895235 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895257 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895261 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895264 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895267 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895270 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895272 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895275 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895278 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895280 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895283 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895285 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895288 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895290 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895293 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895295 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895298 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895301 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.897825 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895303 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895306 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895308 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895311 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895315 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895318 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895320 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895323 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895325 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895328 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895330 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895334 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895338 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895341 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895345 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895348 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895351 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895353 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895356 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.898526 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895358 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895361 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895366 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895368 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895371 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895373 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895376 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895378 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895381 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895383 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895386 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895389 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895391 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895394 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895396 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895400 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895915 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895928 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895932 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.899084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895935 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895938 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895940 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895943 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895946 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895950 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895952 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895955 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895958 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895960 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895963 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895965 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895968 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895970 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895973 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895976 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895979 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895981 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895984 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.899609 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895987 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895989 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895992 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895994 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895997 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.895999 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896002 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896006 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896009 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896011 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896015 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896018 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896020 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896023 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896025 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896028 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896030 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896033 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896035 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.900044 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896038 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896040 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896043 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896045 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896048 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896050 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896053 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896055 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896058 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896060 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896063 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896066 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896069 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896071 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896073 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896076 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896078 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896080 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896083 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896085 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.900516 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896088 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896090 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896092 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896095 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896098 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896101 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896103 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896106 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896108 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896111 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896113 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896116 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896118 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896123 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896127 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896130 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896133 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896135 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896138 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.900973 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896140 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896143 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896145 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896148 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896151 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896154 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896158 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896161 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896164 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896167 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896169 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896172 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896174 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896177 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896180 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896183 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896186 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896189 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896192 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.901428 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896194 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896198 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896200 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896203 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896205 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.896208 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897372 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897382 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897389 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897394 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897399 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897402 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897406 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897410 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897414 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897417 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897420 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897424 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897427 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897431 2580 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897434 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897437 2580 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897440 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:08:07.901868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897443 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897446 2580 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897449 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897452 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897455 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897459 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897464 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897466 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897469 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897473 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897476 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897479 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897482 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897485 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897488 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897492 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897495 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897498 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897501 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897504 2580 flags.go:64] FLAG: --enable-server="true" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897507 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897511 2580 flags.go:64] FLAG: --event-burst="100" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897514 2580 flags.go:64] FLAG: --event-qps="50" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897518 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:08:07.902405 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897521 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897524 2580 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897528 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897532 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897535 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897538 2580 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897541 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897543 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897546 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897549 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897552 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897555 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897557 2580 flags.go:64] FLAG: --feature-gates="" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897562 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897565 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897568 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897571 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897574 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897577 2580 flags.go:64] FLAG: --help="false" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897580 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897583 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897587 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897589 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:08:07.903003 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897593 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897596 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897599 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897602 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897604 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897607 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897610 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897613 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897616 2580 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897619 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897622 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897625 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897627 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897631 2580 flags.go:64] FLAG: --lock-file="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897633 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897636 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897640 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897645 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897648 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897651 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897654 2580 flags.go:64] FLAG: --logging-format="text" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897657 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897661 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897664 2580 flags.go:64] FLAG: --manifest-url="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897667 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:08:07.903542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897671 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897674 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897678 2580 flags.go:64] FLAG: --max-pods="110" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897681 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897684 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897687 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897690 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897693 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897697 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897699 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897705 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897709 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897712 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897715 2580 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897719 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897721 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897724 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897727 2580 flags.go:64] FLAG: --port="10250" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897730 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897733 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00bec3189046f4224" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897737 2580 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897740 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897743 2580 flags.go:64] FLAG: --register-node="true" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897746 2580 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897753 2580 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897756 2580 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:08:07.904135 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897759 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897762 2580 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897766 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897769 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897772 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897774 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897777 2580 flags.go:64] FLAG: --runonce="false" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897780 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897783 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897786 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897788 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897791 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897794 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897797 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897800 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897803 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897806 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897809 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897812 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897815 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897820 2580 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897823 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897828 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897831 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897834 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:08:07.904728 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897838 2580 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897841 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897843 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897847 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897850 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897853 2580 flags.go:64] FLAG: --v="2" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897857 2580 flags.go:64] FLAG: --version="false" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897861 2580 flags.go:64] FLAG: --vmodule="" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897865 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.897870 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899650 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899664 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899667 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899671 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899674 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899676 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899679 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899682 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899685 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899687 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899690 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.905309 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899692 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899695 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899698 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899700 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899703 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899705 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899708 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899711 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899713 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899716 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899719 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899721 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899724 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899726 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899729 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899732 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899734 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899737 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899740 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899743 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.906022 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899750 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899753 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899757 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899760 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899763 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899766 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899768 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899771 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899774 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899776 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899779 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899782 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899784 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899787 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899789 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899792 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899794 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899797 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899800 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899802 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.906948 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899805 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899807 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899810 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899813 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899815 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899818 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899820 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899823 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899827 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899832 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899835 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899838 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899841 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899844 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899848 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899850 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899855 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899858 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899861 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.907770 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899863 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899866 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899868 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899871 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899873 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899876 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899879 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899881 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899884 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899886 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899889 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899891 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899894 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899897 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899900 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899903 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899906 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899908 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899911 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899914 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.908669 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899917 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899919 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899922 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899924 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899927 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899929 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899932 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899934 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899937 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899939 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899942 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899944 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899946 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899949 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899952 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.899954 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.909531 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.899985 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900031 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900034 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900039 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900042 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900045 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900048 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900051 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900055 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900058 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900061 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900064 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900066 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900069 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900071 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900074 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900076 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.910129 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900079 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900081 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900084 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900087 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900089 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900092 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900094 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900097 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900099 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900102 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900105 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900107 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900110 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900112 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900115 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900117 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900119 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900122 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900125 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.910603 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900128 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900132 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900135 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900137 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900140 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900143 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900145 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900148 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900152 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900155 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900158 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900161 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900163 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900166 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900168 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900170 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900173 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900176 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900178 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.911300 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900181 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900183 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900186 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900191 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900194 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900197 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900199 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900203 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900205 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900208 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900210 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900213 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900215 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900218 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900220 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900223 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900225 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900228 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900230 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.912057 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900235 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900251 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900254 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900257 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900259 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900262 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900266 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900268 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900271 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900273 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900276 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900278 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900281 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900283 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900286 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900290 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900293 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900295 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900298 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.912569 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900300 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900303 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900305 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900308 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900310 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900313 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900315 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900318 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900322 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900326 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.900328 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.907894 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.907914 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.907937 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.907944 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908026 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908035 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908041 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908045 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908050 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.913125 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908054 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908058 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908062 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908067 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908071 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908075 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908079 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908083 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908088 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908092 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908096 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908100 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908105 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908110 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908114 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908118 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908122 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908127 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908131 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908136 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.913624 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908140 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908144 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908149 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908153 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908158 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908164 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908168 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908173 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908178 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908182 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908186 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908190 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908195 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908199 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908203 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908207 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908211 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908216 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908220 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908224 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.914084 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908228 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908232 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908236 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908261 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908265 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908269 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908273 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908277 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908281 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908285 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908289 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908293 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908297 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908302 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908306 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908310 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908315 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908322 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.914698 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908329 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908335 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908340 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908344 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908348 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908352 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908356 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908361 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908365 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908369 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908373 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908377 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908381 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908385 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908390 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908394 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908398 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908402 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908406 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908411 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.915198 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908414 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908419 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908423 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908427 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908433 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908438 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908443 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908448 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908453 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908457 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908461 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908465 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908470 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908475 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908479 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908484 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908488 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908492 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908496 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.915673 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908501 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908505 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908509 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908513 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.908556 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908802 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908810 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908815 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908819 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908824 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908828 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908832 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908836 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908840 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908844 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908849 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.916141 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908853 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908857 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908861 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908865 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908869 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908874 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908879 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908885 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908890 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908894 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908898 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908907 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908912 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908916 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908920 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908924 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908928 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908932 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.916543 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908936 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908940 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908944 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908949 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908953 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908957 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908961 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908965 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908969 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908973 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908977 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908981 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908986 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908990 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908994 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.908999 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909003 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909007 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909011 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909015 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.916978 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909019 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909023 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909028 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909032 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909036 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909041 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909046 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909050 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909054 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909058 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909062 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909066 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909070 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909075 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909079 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909083 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909087 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909091 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909095 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909099 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.917466 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909103 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909108 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909112 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909116 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909121 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909125 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909130 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909134 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909138 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909142 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909146 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909153 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909159 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909164 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909168 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909172 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909176 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909180 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909185 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.917909 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909189 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909193 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909197 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909201 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909205 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909209 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909214 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909218 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909222 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909226 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909230 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909234 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909254 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909259 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909263 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909267 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.918358 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909271 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.909311 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909360 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909366 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909371 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909375 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909385 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909391 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909396 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909401 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909405 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909409 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909413 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909418 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909422 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:07.918719 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909426 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909431 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909435 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909440 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909444 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909448 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909452 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909456 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909460 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909464 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909468 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909472 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909476 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909481 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909485 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909489 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909493 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909497 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909502 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909506 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:07.919059 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909510 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909515 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909519 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909525 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909529 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909533 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909537 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909541 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909545 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909549 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909553 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909557 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909562 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909566 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909571 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909575 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909579 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909585 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909589 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909593 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:07.919554 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909597 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909601 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909608 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909613 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909622 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909626 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909630 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909634 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909639 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909643 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909647 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909651 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909655 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909660 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909668 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909674 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909678 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909682 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:07.920009 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909686 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909691 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909695 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909699 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909703 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909707 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909711 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909715 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909720 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909724 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909728 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909733 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909737 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909741 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909745 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909751 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909755 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909759 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909763 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909767 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:07.920450 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909772 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909776 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909781 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909785 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909789 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909795 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909799 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909803 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909809 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909813 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909817 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909821 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909826 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:07.909830 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.909900 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.912966 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.914085 2580 server.go:1014] "Starting client certificate rotation" Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.914181 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:08:07.920887 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.914215 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:08:07.921315 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.916400 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:08:07.921315 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.917384 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:08:07.927752 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.927738 2580 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:08:07.931543 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.931530 2580 log.go:25] "Validated CRI v1 image API" Nov 02 21:08:07.932369 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.932345 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:08:07.935880 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.935857 2580 fs.go:135] Filesystem UUIDs: map[5001333b-80af-4f93-9425-cce0ac96e88f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b023e6e7-bcf4-424e-8270-371d20d8db7c:/dev/nvme0n1p3] Nov 02 21:08:07.935927 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.935880 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:08:07.941690 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.941585 2580 manager.go:217] Machine: {Timestamp:2026-11-02 21:08:07.940126528 +0000 UTC m=+0.376128443 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201118 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec218698333d06c45c3c8121c41a55a0 SystemUUID:ec218698-333d-06c4-5c3c-8121c41a55a0 BootID:875c9d7b-914f-4da0-aa3f-a415de742a21 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:b8:e1:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:b8:e1:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:f9:27:bb:59:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:08:07.941690 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.941683 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:08:07.941827 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.941752 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:08:07.945148 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945123 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:08:07.945349 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945150 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-1.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:08:07.945435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945353 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:08:07.945435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945365 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:08:07.945435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945402 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:08:07.945435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.945422 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:08:07.946778 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.946765 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:08:07.951439 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.951426 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:08:07.951563 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.951553 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:08:07.952430 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952419 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:08:07.952482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952436 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:08:07.952482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952451 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:08:07.952482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952465 2580 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:08:07.952482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952475 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:08:07.952795 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.952782 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:08:07.955348 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.955329 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:08:07.961480 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.961467 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:08:07.962419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962406 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:08:07.962454 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962437 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:08:07.962605 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962595 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:08:07.962643 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962612 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:08:07.962643 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962620 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:08:07.962643 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962628 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:08:07.962643 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962636 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:08:07.962643 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.962642 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:08:07.963383 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963375 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:08:07.963413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963385 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:08:07.963413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963392 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:08:07.963413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963398 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:08:07.963413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963405 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:08:07.963413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.963413 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:08:07.964143 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.964134 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:08:07.964175 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.964146 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:08:07.967831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.967817 2580 server.go:1263] "Started kubelet" Nov 02 21:08:07.967961 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.967913 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:08:07.968015 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.967970 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:08:07.968015 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.967990 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:08:07.968511 ip-10-0-137-1 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:08:07.969563 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.969543 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:08:07.971122 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.971108 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:08:07.975731 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:07.975693 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:08:07.977331 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.977271 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:08:07.977972 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.977957 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:08:07.978092 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:07.978057 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:08:07.978811 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.978793 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:08:07.978811 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.978812 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:08:07.978996 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.978982 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:08:07.978996 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.978996 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:08:07.979093 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979027 2580 factory.go:55] Registering systemd factory Nov 02 21:08:07.979093 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979043 2580 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:08:07.979228 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979214 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:08:07.979302 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979261 2580 factory.go:153] Registering CRI-O factory Nov 02 21:08:07.979302 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979275 2580 factory.go:223] Registration of the crio container factory successfully Nov 02 21:08:07.979406 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979314 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:08:07.979406 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979331 2580 factory.go:103] Registering Raw factory Nov 02 21:08:07.979406 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.979345 2580 manager.go:1199] Started watching for new ooms in manager Nov 02 21:08:07.980379 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.980365 2580 manager.go:319] Starting recovery of all containers Nov 02 21:08:07.991377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.991359 2580 manager.go:324] Recovery completed Nov 02 21:08:07.995708 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.995693 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:07.997543 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.997527 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:07.997608 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.997556 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:07.997608 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.997568 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:07.998455 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.998441 2580 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:08:07.998455 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.998452 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:08:07.998612 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.998467 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:08:07.998612 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.998492 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:07.999915 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.999900 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:07.999977 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.999921 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:07.999977 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:07.999933 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.000714 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.000700 2580 policy_none.go:50] "Start" Nov 02 21:08:08.000714 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.000716 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:08:08.000833 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.000726 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:08:08.002695 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.002682 2580 policy_none.go:44] "Start" Nov 02 21:08:08.038894 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.038758 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:08:08.043206 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.043194 2580 manager.go:341] "Starting Device Plugin manager" Nov 02 21:08:08.043883 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.043867 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:08:08.043883 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.043883 2580 server.go:87] "Starting device plugin registration server" Nov 02 21:08:08.044077 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.044066 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:08:08.044139 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.044076 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:08:08.044197 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.044183 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:08:08.044292 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.044279 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:08:08.044292 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.044289 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:08:08.045216 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.045202 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:08:08.045308 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.045255 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:08:08.112975 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.112950 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-1.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:08:08.114938 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.113049 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-1.us-west-2.compute.internal.18e45686a963b4cc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-1.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-1.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:08:07.967790284 +0000 UTC m=+0.403792192,LastTimestamp:2026-11-02 21:08:07.967790284 +0000 UTC m=+0.403792192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-1.us-west-2.compute.internal,}" Nov 02 21:08:08.115586 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.115558 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:08:08.116674 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.116655 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:08:08.116674 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.116675 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:08:08.116830 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.116691 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:08:08.116830 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.116720 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:08:08.144560 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.144509 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:08.145230 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.145216 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:08.145319 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.145255 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:08.145319 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.145266 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.145319 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.145288 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.162266 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.162235 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-1.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:08:08.190172 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.190149 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:08:08.190295 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.190191 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2qms" Nov 02 21:08:08.216946 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.216917 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal"] Nov 02 21:08:08.217021 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.216977 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:08.221306 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.221291 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:08.221386 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.221318 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:08.221386 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.221333 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.223482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.223470 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:08.224515 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.224268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.224515 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.224328 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:08.225942 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.225924 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:08.226024 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.225951 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:08.226024 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.225974 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:08.226024 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.225983 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.226024 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.225955 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:08.226143 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.226039 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.228206 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.228193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.228293 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.228216 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:08.228950 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.228929 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:08.228950 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.228951 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:08.229053 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.228963 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:08.255729 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.255706 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.258096 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.258079 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.263268 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.263253 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:08.264219 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.264207 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:08.264787 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.264768 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:08.358849 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.358828 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:08.360004 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.359989 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.360078 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:08.360013 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-1.us-west-2.compute.internal\": node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:08:08.378249 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.378222 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.380411 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.380397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.380456 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.380420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.380456 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.380437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.436607 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.436571 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:08:08.436669 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.436650 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480792 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480868 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480855 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480971 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480871 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.480971 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.480886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.489897 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.489883 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:08:08.558032 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.558014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:08.561176 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:08.561158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:08:10.063388 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.063367 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:08:10.063812 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.063461 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:08:10.063812 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.063508 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:08:10.063812 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.063529 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:08:10.063812 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.063530 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab303f828c2034a3dad08796677c6e91-0df893518692fff8.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-1.us-west-2.compute.internal.18e45686ab29ae90\": read tcp 10.0.137.1:44192->18.227.67.248:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-1.us-west-2.compute.internal.18e45686ab29ae90 default 5528 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-1.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-1.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-1.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:08:07 +0000 UTC,LastTimestamp:2026-11-02 21:08:08.221304999 +0000 UTC m=+0.657306907,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-1.us-west-2.compute.internal,}" Nov 02 21:08:10.103278 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.103262 2580 apiserver.go:52] "Watching apiserver" Nov 02 21:08:10.127996 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.127979 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:08:10.344542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.344483 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:03:09 +0000 UTC" deadline="2028-04-01 12:59:49.633027448 +0000 UTC" Nov 02 21:08:10.344542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.344511 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12375h51m39.288518924s" Nov 02 21:08:10.349731 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.349712 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:08:10.350008 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.349993 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xm28p","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal","openshift-image-registry/node-ca-8spgj","openshift-network-diagnostics/network-check-target-pt5zs","openshift-network-operator/iptables-alerter-fzp9j","openshift-ovn-kubernetes/ovnkube-node-rsvj8","kube-system/konnectivity-agent-vbbpt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp","openshift-dns/node-resolver-9jbsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-kmm87","openshift-cluster-node-tuning-operator/tuned-8pk9f","openshift-multus/multus-ssnrt","openshift-multus/network-metrics-daemon-xk8mq"] Nov 02 21:08:10.352376 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.352363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.352437 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.352423 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:10.354341 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.354328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.356378 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.356351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:10.356466 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.356400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:10.358373 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.358356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.360528 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.360509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.362674 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.362658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.364893 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.364878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.367166 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.367149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.370372 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.369868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.372386 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.372365 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.375458 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.375441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.377567 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.377549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.377639 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.377598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:10.402966 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.402932 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.403049 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.402939 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.403049 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.403037 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:08:10.406477 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.406464 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.406613 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.406599 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.406667 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.406655 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:08:10.408931 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.408917 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:08:10.409021 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.408992 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.409067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.409020 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.409067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.409014 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:08:10.409250 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.409239 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:08:10.410444 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.410427 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:08:10.430082 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.430068 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:08:10.434773 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.434835 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.434835 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.434835 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jds\" (UniqueName: \"kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.434835 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2sd\" (UniqueName: \"kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.435027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.435027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.435027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.435027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzks\" (UniqueName: \"kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.434969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.435259 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gcx\" (UniqueName: \"kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.435595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.435981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqq6\" (UniqueName: \"kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thd2l\" (UniqueName: \"kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fdm\" (UniqueName: \"kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.436762 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsrf\" (UniqueName: \"kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk7q6\" (UniqueName: \"kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:10.437227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.437636 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.436775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.453638 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453620 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\"" Nov 02 21:08:10.453854 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453826 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\"" Nov 02 21:08:10.453924 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453891 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\"" Nov 02 21:08:10.453924 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453906 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:08:10.454032 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453922 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\"" Nov 02 21:08:10.454032 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.453934 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\"" Nov 02 21:08:10.455326 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455310 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:08:10.455387 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455358 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:08:10.455387 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455363 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\"" Nov 02 21:08:10.455472 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455390 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\"" Nov 02 21:08:10.455472 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455464 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\"" Nov 02 21:08:10.455538 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.455473 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\"" Nov 02 21:08:10.498671 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.498648 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:08:10.498743 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.498705 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h5src" Nov 02 21:08:10.504576 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504558 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504625 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504635 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504631 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504659 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504663 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.504670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504671 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:08:10.504869 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504626 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.504869 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:10.504869 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:08:10.504869 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504675 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:08:10.504869 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.504716 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:08:10.537931 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.537915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.537931 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.537936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.537962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.537976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.537990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.538067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538056 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538066 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538071 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538115 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538184 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gcx\" (UniqueName: \"kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538247 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538264 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.538427 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.538452 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.538501 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:11.038483364 +0000 UTC m=+2.324774545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538588 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.539182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538760 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538568 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538794 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538758 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538817 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqq6\" (UniqueName: \"kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538314 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.539978 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thd2l\" (UniqueName: \"kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539037 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fdm\" (UniqueName: \"kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539074 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.540757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsrf\" (UniqueName: \"kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539298 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.538934 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539365 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539391 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539513 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539572 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7q6\" (UniqueName: \"kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539622 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.541554 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539635 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539620 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539575 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539211 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539672 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539708 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539712 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539717 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539723 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539792 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539818 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539832 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539825 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539856 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.544419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539944 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.539968 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.539988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.540010 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:11.039996115 +0000 UTC m=+2.326287301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540015 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540201 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jds\" (UniqueName: \"kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2sd\" (UniqueName: \"kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540271 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.544872 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540323 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540365 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzks\" (UniqueName: \"kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540556 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.540645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.541042 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.541058 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.541309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.545360 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.541559 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545823 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.541616 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.545823 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.542687 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.545823 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.542790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.545823 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.542791 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.545823 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.543139 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.591724 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.591696 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:10.591724 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.591721 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:10.591876 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.591733 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:10.591876 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:10.591787 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:11.091772327 +0000 UTC m=+2.378063520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:10.596122 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596049 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7q6\" (UniqueName: \"kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.596264 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596202 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2sd\" (UniqueName: \"kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.596343 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596285 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gcx\" (UniqueName: \"kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.596431 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsrf\" (UniqueName: \"kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.596649 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596621 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fdm\" (UniqueName: \"kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:10.596752 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.596677 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqq6\" (UniqueName: \"kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.597251 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.597223 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thd2l\" (UniqueName: \"kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.597425 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.597403 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jds\" (UniqueName: \"kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.598297 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.598277 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzks\" (UniqueName: \"kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.676413 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.676388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:08:10.694789 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.694768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:08:10.699716 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.699700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:08:10.730583 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.730564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:10.730666 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.730653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:10.738507 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.738492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:08:10.746407 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.746380 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e83f034a10ddc04c4aab6039382daa.slice/crio-3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472 WatchSource:0}: Error finding container 3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472: Status 404 returned error can't find the container with id 3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472 Nov 02 21:08:10.747008 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.746984 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d04d90_6021_4329_b47d_a41847969eef.slice/crio-635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634 WatchSource:0}: Error finding container 635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634: Status 404 returned error can't find the container with id 635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634 Nov 02 21:08:10.747486 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.747469 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e07eb1f_6b9f_4626_ab55_efb294a3d136.slice/crio-917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1 WatchSource:0}: Error finding container 917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1: Status 404 returned error can't find the container with id 917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1 Nov 02 21:08:10.748150 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.748123 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedbdf8e991546e8774cd514478921e1.slice/crio-fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4 WatchSource:0}: Error finding container fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4: Status 404 returned error can't find the container with id fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4 Nov 02 21:08:10.749975 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.749845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:08:10.750289 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.750265 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d0c582_d441_4746_8619_d43d8f4099fe.slice/crio-da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3 WatchSource:0}: Error finding container da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3: Status 404 returned error can't find the container with id da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3 Nov 02 21:08:10.751782 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.751772 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:08:10.757996 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.757968 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcdbaab_36bb_4f9a_9008_4e21d230deea.slice/crio-b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1 WatchSource:0}: Error finding container b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1: Status 404 returned error can't find the container with id b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1 Nov 02 21:08:10.776192 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.776175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ssnrt" Nov 02 21:08:10.782846 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:10.782831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:08:10.783980 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.783963 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdffb38_5e8e_4484_a77a_abaea4c5aca4.slice/crio-7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5 WatchSource:0}: Error finding container 7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5: Status 404 returned error can't find the container with id 7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5 Nov 02 21:08:10.789874 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:10.789857 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf62419_1dfd_4003_937c_6f3a835bd88e.slice/crio-021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec WatchSource:0}: Error finding container 021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec: Status 404 returned error can't find the container with id 021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec Nov 02 21:08:11.044228 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.044176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:11.044324 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.044225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:11.044324 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.044316 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:11.044424 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.044361 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:12.044345327 +0000 UTC m=+3.330636508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:11.044424 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.044405 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:11.044501 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.044446 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:12.044434819 +0000 UTC m=+3.330726004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:11.144633 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.144611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:11.145123 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.144700 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:11.145123 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.144713 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:11.145123 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.144722 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:11.145123 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:11.144757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:12.144747156 +0000 UTC m=+3.431038338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:11.270538 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.270496 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1"} Nov 02 21:08:11.271585 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.271566 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbbpt" event={"ID":"04d04d90-6021-4329-b47d-a41847969eef","Type":"ContainerStarted","Data":"635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634"} Nov 02 21:08:11.273310 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.273268 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerStarted","Data":"3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472"} Nov 02 21:08:11.274796 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.274765 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"aedbdf8e991546e8774cd514478921e1","Type":"ContainerStarted","Data":"fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4"} Nov 02 21:08:11.276200 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.276181 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerStarted","Data":"021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec"} Nov 02 21:08:11.277158 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.277140 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssnrt" event={"ID":"fbdffb38-5e8e-4484-a77a-abaea4c5aca4","Type":"ContainerStarted","Data":"7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5"} Nov 02 21:08:11.278079 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.278057 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jbsf" event={"ID":"efcdbaab-36bb-4f9a-9008-4e21d230deea","Type":"ContainerStarted","Data":"b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1"} Nov 02 21:08:11.278896 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.278880 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" event={"ID":"61d0c582-d441-4746-8619-d43d8f4099fe","Type":"ContainerStarted","Data":"da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3"} Nov 02 21:08:11.436497 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.436447 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:11.457975 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.457946 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:11.468029 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:11.468007 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4238a8dd_cf04_4f35_bb82_fce7996270a4.slice/crio-2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67 WatchSource:0}: Error finding container 2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67: Status 404 returned error can't find the container with id 2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67 Nov 02 21:08:11.502075 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.502036 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:03:10 +0000 UTC" deadline="2028-06-06 02:40:28.797127022 +0000 UTC" Nov 02 21:08:11.502075 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.502071 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13949h32m17.295061447s" Nov 02 21:08:11.593539 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:11.593520 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c7a3bc_93b3_4c1a_8c7f_8a36b7ef55f2.slice/crio-4737b2d774099068c52f9cb3d57f25748ca68a991e7acec1c0d5951025b0816d WatchSource:0}: Error finding container 4737b2d774099068c52f9cb3d57f25748ca68a991e7acec1c0d5951025b0816d: Status 404 returned error can't find the container with id 4737b2d774099068c52f9cb3d57f25748ca68a991e7acec1c0d5951025b0816d Nov 02 21:08:11.594792 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:11.594773 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018ea93d_84d6_4e86_98ce_e376674b7a12.slice/crio-2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be WatchSource:0}: Error finding container 2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be: Status 404 returned error can't find the container with id 2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be Nov 02 21:08:11.684461 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:11.684439 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:12.052932 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.052868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:12.052932 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.052930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:12.053170 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.053081 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:12.053170 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.053137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:14.053120092 +0000 UTC m=+5.339411287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:12.053284 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.053202 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:12.053284 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.053236 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:14.053225639 +0000 UTC m=+5.339516828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:12.153512 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.153476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:12.154002 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.153627 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:12.154002 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.153650 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:12.154002 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.153662 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:12.154002 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.153717 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:14.153701096 +0000 UTC m=+5.439992290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:12.267009 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.266979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:12.267171 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.267087 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:12.267171 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.267151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:12.267281 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.267203 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:12.267281 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.267240 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:12.267384 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:12.267280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:12.297529 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.297496 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzp9j" event={"ID":"018ea93d-84d6-4e86-98ce-e376674b7a12","Type":"ContainerStarted","Data":"2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be"} Nov 02 21:08:12.305062 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.305005 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" event={"ID":"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2","Type":"ContainerStarted","Data":"4737b2d774099068c52f9cb3d57f25748ca68a991e7acec1c0d5951025b0816d"} Nov 02 21:08:12.307418 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.307391 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8spgj" event={"ID":"4238a8dd-cf04-4f35-bb82-fce7996270a4","Type":"ContainerStarted","Data":"2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67"} Nov 02 21:08:12.503092 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.503035 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:03:10 +0000 UTC" deadline="2028-05-26 15:38:04.82876186 +0000 UTC" Nov 02 21:08:12.503092 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:12.503070 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13698h29m52.325696272s" Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.068121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.068190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.068311 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.068360 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.068345415 +0000 UTC m=+9.354636599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.068423 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:14.068500 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.068457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.068445913 +0000 UTC m=+9.354737095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:14.169377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.169156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:14.169377 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.169291 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:14.169377 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.169304 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:14.169377 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.169312 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:14.169377 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.169355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.169341496 +0000 UTC m=+9.455632678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.266859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.267003 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.267070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.267213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:14.267261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:14.267372 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:14.267325 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:15.321298 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:15.321064 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="1c9423bd8261afdb571a485668a302083e835b625b9b98ed664370b46a15cf03" exitCode=0 Nov 02 21:08:15.321666 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:15.321363 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"1c9423bd8261afdb571a485668a302083e835b625b9b98ed664370b46a15cf03"} Nov 02 21:08:15.335369 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:15.335268 2580 generic.go:356] "Generic (PLEG): container finished" podID="61e83f034a10ddc04c4aab6039382daa" containerID="9b2a1cc1da1138b2856dd99f3f8420ecc822e08765fe6ba3debb177ceac01f1a" exitCode=0 Nov 02 21:08:15.335369 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:15.335305 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerDied","Data":"9b2a1cc1da1138b2856dd99f3f8420ecc822e08765fe6ba3debb177ceac01f1a"} Nov 02 21:08:16.267506 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.267480 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:16.267665 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:16.267591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:16.267742 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.267668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:16.267797 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:16.267749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:16.267852 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.267808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:16.267902 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:16.267875 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:16.339521 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.339483 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"aedbdf8e991546e8774cd514478921e1","Type":"ContainerStarted","Data":"a1bf97d65886c9b9f988364a438aadeb5135c297690ef2c8b3ee3193d17ce8af"} Nov 02 21:08:16.344452 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.344422 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbbpt" event={"ID":"04d04d90-6021-4329-b47d-a41847969eef","Type":"ContainerStarted","Data":"8e49ca8ff62e01af25224c6e556f948c8022f33e5a3020026a9886f8fc43490f"} Nov 02 21:08:16.348051 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.348030 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerStarted","Data":"c6e9b9d823f782260537ff1a3d8e447a82739560e6340666cadd161f289cc0c5"} Nov 02 21:08:16.445499 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.445413 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" podStartSLOduration=7.445399905 podStartE2EDuration="7.445399905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:16.444597022 +0000 UTC m=+7.730888224" watchObservedRunningTime="2026-11-02 21:08:16.445399905 +0000 UTC m=+7.731691108" Nov 02 21:08:16.657599 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:16.657308 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" podStartSLOduration=7.657287528 podStartE2EDuration="7.657287528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:16.551603848 +0000 UTC m=+7.837895053" watchObservedRunningTime="2026-11-02 21:08:16.657287528 +0000 UTC m=+7.943578732" Nov 02 21:08:17.209798 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:17.209751 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:17.355013 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:17.354987 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8spgj" event={"ID":"4238a8dd-cf04-4f35-bb82-fce7996270a4","Type":"ContainerStarted","Data":"8854579fd48fac566e6a78e113e5d678a86ba5d2229845095adfc0d14b41025d"} Nov 02 21:08:17.474265 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:17.473742 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbbpt" podStartSLOduration=4.058598341 podStartE2EDuration="8.473722531s" totalImagesPullingTime="4.41512419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:16.65753273 +0000 UTC m=+7.943823943" watchObservedRunningTime="2026-11-02 21:08:17.473722531 +0000 UTC m=+8.760013746" Nov 02 21:08:18.101445 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.101370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:18.101608 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.101446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:18.101608 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.101587 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:18.101716 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.101651 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:26.101633539 +0000 UTC m=+17.387924723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:18.101780 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.101727 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:18.101780 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.101762 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:26.101750409 +0000 UTC m=+17.388041593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:18.202729 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.202535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:18.202729 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.202685 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:18.202729 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.202707 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:18.202729 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.202719 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:18.203051 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.202774 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:26.202758424 +0000 UTC m=+17.489049611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:18.266995 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.266944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:18.267169 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.267090 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:18.267169 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.267153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:18.267272 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.267200 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:18.267272 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.267236 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:18.267378 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:18.267276 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:18.358120 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:18.358028 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" event={"ID":"61d0c582-d441-4746-8619-d43d8f4099fe","Type":"ContainerStarted","Data":"6131cfb7e617cc36b58f71369f3ae64636a83d83c662125d5f0736de2ebd1886"} Nov 02 21:08:19.360807 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:19.360779 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ssnrt" event={"ID":"fbdffb38-5e8e-4484-a77a-abaea4c5aca4","Type":"ContainerStarted","Data":"dfac9ac4c17778c53887635f43e6d483cdebd7763ff7dcc1d8d07b1d25bfbb04"} Nov 02 21:08:19.363146 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:19.363119 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9jbsf" event={"ID":"efcdbaab-36bb-4f9a-9008-4e21d230deea","Type":"ContainerStarted","Data":"1b2cf23b7ab18f0eb8dd5057765a08711db089b1fcb5fe22944277598c392610"} Nov 02 21:08:19.502061 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:19.502001 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:08:19.512010 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:19.511938 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8spgj" podStartSLOduration=4.919759485 podStartE2EDuration="10.511925898s" totalImagesPullingTime="5.592166413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:17.474832077 +0000 UTC m=+8.761123282" watchObservedRunningTime="2026-11-02 21:08:19.511925898 +0000 UTC m=+10.798217099" Nov 02 21:08:19.617488 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:19.617438 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ssnrt" podStartSLOduration=2.508073409 podStartE2EDuration="10.617423926s" totalImagesPullingTime="8.109350517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:19.512705083 +0000 UTC m=+10.798996285" watchObservedRunningTime="2026-11-02 21:08:19.617423926 +0000 UTC m=+10.903715129" Nov 02 21:08:20.198657 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.198397 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:08:19.502021613Z","UUID":"9b79314f-fcd8-4d34-85c8-0c0d705c014c","Handler":null,"Name":"","Endpoint":""} Nov 02 21:08:20.200057 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.200036 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:08:20.200164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.200071 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:08:20.265689 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.265655 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:20.266796 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.266776 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:20.266922 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.266825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:20.266994 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:20.266938 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:20.267053 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.267000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:20.267113 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:20.267090 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:20.267157 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.267150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:20.267226 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:20.267207 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:20.369119 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.369078 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" event={"ID":"61d0c582-d441-4746-8619-d43d8f4099fe","Type":"ContainerStarted","Data":"1b33224a3b4bd41c0f22c684b45da4eea675abd27c377442eab9d06bfe4ac162"} Nov 02 21:08:20.371066 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.371008 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="24abd1b375a8af4eada026668df62933526a613ad135a1266911278a31a59328" exitCode=0 Nov 02 21:08:20.371203 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.371102 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"24abd1b375a8af4eada026668df62933526a613ad135a1266911278a31a59328"} Nov 02 21:08:20.372318 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.372298 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:08:20.373918 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:20.373360 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9jbsf" podStartSLOduration=2.989543197 podStartE2EDuration="11.373345553s" totalImagesPullingTime="8.383802356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:19.617757709 +0000 UTC m=+10.904048911" watchObservedRunningTime="2026-11-02 21:08:20.373345553 +0000 UTC m=+11.659636756" Nov 02 21:08:21.375899 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:21.375546 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" event={"ID":"61d0c582-d441-4746-8619-d43d8f4099fe","Type":"ContainerStarted","Data":"59c865b97b8a21f09161c50a6034d0c3457f1192a26568c9264f564f8c02d749"} Nov 02 21:08:21.377140 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:21.377117 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzp9j" event={"ID":"018ea93d-84d6-4e86-98ce-e376674b7a12","Type":"ContainerStarted","Data":"6fb1f1a18b2b87dd26c17818a3c77e9a2965ceb0c2baf90831300e52c604881b"} Nov 02 21:08:21.590081 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:21.590020 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fzp9j" podStartSLOduration=4.31676372 podStartE2EDuration="12.590003046s" totalImagesPullingTime="8.273239326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:21.589187681 +0000 UTC m=+12.875478884" watchObservedRunningTime="2026-11-02 21:08:21.590003046 +0000 UTC m=+12.876294249" Nov 02 21:08:21.590439 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:21.590410 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" podStartSLOduration=2.786821738 podStartE2EDuration="12.590401641s" totalImagesPullingTime="9.803579903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:21.481986042 +0000 UTC m=+12.768277244" watchObservedRunningTime="2026-11-02 21:08:21.590401641 +0000 UTC m=+12.876692844" Nov 02 21:08:22.267487 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.267411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:22.267487 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.267466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:22.267487 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.267478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:22.267733 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:22.267562 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:22.267733 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:22.267634 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:22.267733 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:22.267705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:22.382014 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.381987 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="535a4c60b5a8c6028889873a3591bc4fbc1e1ac58badb15fbd54c43636e50f41" exitCode=0 Nov 02 21:08:22.382667 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.382069 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"535a4c60b5a8c6028889873a3591bc4fbc1e1ac58badb15fbd54c43636e50f41"} Nov 02 21:08:22.383903 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:22.383878 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" event={"ID":"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2","Type":"ContainerStarted","Data":"b9c12efd253c7b57e6b1577a22e03d6677bbfec9e804b818793ac78c52767e53"} Nov 02 21:08:24.267779 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.267478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:24.267779 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.267494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:24.268256 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:24.267805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:24.268256 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.267516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:24.268256 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:24.267878 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:24.268256 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:24.267960 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:24.390023 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.389994 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="e5efced5a740c8ad1d147972842c300bdd4c5de22deefa8fc3bbaa6eaa80318f" exitCode=0 Nov 02 21:08:24.390158 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.390036 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"e5efced5a740c8ad1d147972842c300bdd4c5de22deefa8fc3bbaa6eaa80318f"} Nov 02 21:08:24.501397 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:24.501352 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" podStartSLOduration=5.227297646 podStartE2EDuration="15.501340461s" totalImagesPullingTime="10.274042815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:22.600922809 +0000 UTC m=+13.887214011" watchObservedRunningTime="2026-11-02 21:08:24.501340461 +0000 UTC m=+15.787631664" Nov 02 21:08:25.393931 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:25.393906 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"2477cc90217774ed9b17e9260a5963da50176ff83025aa6e4b0ef63a4224c0fc"} Nov 02 21:08:26.162556 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.162365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:26.162840 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.162602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:26.162840 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.162495 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:26.162840 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.162709 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:26.162840 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.162718 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:42.16269686 +0000 UTC m=+33.448988043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:26.162840 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.162750 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:42.162738912 +0000 UTC m=+33.449030093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:26.263134 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.263108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:26.263266 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.263214 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:26.263266 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.263226 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:26.263266 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.263235 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:26.263382 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.263273 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:42.263262623 +0000 UTC m=+33.549553805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:26.267188 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.267166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:26.267309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.267195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:26.267309 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.267269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:26.267309 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.267284 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:26.267420 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.267340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:26.267488 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:26.267429 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:26.398501 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.398470 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerStarted","Data":"79517b0601ba490bcdc91a160caa674ab723bdc131d1166335acfa6be05918fb"} Nov 02 21:08:26.404288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.401735 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"66de255c89e335329446365743c9cba6de485295a3bac65f277c0a18baf39abd"} Nov 02 21:08:26.404288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.401886 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"5b9f47fb8218f485ef6eacb40f40115aa6ba648fe21547bb0ecf864fe496b079"} Nov 02 21:08:26.404288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.401979 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"63286c22f38528bcf5458d6b1d30e641f9fb534d6fb5886447815f892239cc86"} Nov 02 21:08:26.404288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.402047 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"af561973a3043f1db531ec54a3e59e8e67d18571552fd1c0b11619b67557a655"} Nov 02 21:08:26.404288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:26.402111 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"6bd964f50d88f7a48259bc859a0171cb1dd43626c2d19e54490291d45f033527"} Nov 02 21:08:27.406417 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:27.406384 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="79517b0601ba490bcdc91a160caa674ab723bdc131d1166335acfa6be05918fb" exitCode=0 Nov 02 21:08:27.406863 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:27.406440 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"79517b0601ba490bcdc91a160caa674ab723bdc131d1166335acfa6be05918fb"} Nov 02 21:08:28.267491 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.267430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:28.267491 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.267460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:28.267491 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.267467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:28.267676 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:28.267538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:28.267676 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:28.267583 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:28.267676 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:28.267656 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:28.411297 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.411269 2580 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="b32e183fa1b65821e3f9f1a8e2970260188f5a68a9d159dad69e150a4d892d43" exitCode=0 Nov 02 21:08:28.411783 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.411338 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerDied","Data":"b32e183fa1b65821e3f9f1a8e2970260188f5a68a9d159dad69e150a4d892d43"} Nov 02 21:08:28.414688 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:28.414670 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"86d37581d203bf88a734a78fd9400430545ba63a2698d6e65302f1df43d99620"} Nov 02 21:08:29.419606 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:29.419571 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmm87" event={"ID":"baf62419-1dfd-4003-937c-6f3a835bd88e","Type":"ContainerStarted","Data":"24c0acab50d7f5d1db6a4656852e7144bfe3fb2c8c75fb277174019841ca681c"} Nov 02 21:08:29.623835 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:29.623796 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmm87" podStartSLOduration=7.8894684470000005 podStartE2EDuration="20.623783797s" totalImagesPullingTime="12.73431535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:29.577001119 +0000 UTC m=+20.863292320" watchObservedRunningTime="2026-11-02 21:08:29.623783797 +0000 UTC m=+20.910074998" Nov 02 21:08:30.267571 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:30.267481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:30.267571 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:30.267499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:30.267571 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:30.267525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:30.267750 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:30.267598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:30.267750 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:30.267650 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:30.267750 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:30.267718 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:30.424860 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:30.424832 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" event={"ID":"7e07eb1f-6b9f-4626-ab55-efb294a3d136","Type":"ContainerStarted","Data":"db15d1c69e2f42870e9a02368a903fea714312d5e6da2637226bb0ffe992aad8"} Nov 02 21:08:30.589380 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:30.589299 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" podStartSLOduration=7.039611052 podStartE2EDuration="21.589287997s" totalImagesPullingTime="14.549676945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:30.588853647 +0000 UTC m=+21.875144850" watchObservedRunningTime="2026-11-02 21:08:30.589287997 +0000 UTC m=+21.875579202" Nov 02 21:08:31.166763 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.166740 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9jbsf_efcdbaab-36bb-4f9a-9008-4e21d230deea/dns-node-resolver/0.log" Nov 02 21:08:31.426411 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.426358 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:08:31.426759 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.426730 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:31.426867 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.426814 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:31.442149 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.442129 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:31.447315 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:31.447301 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:32.206939 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:32.206918 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:08:32.214457 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:32.214436 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8spgj_4238a8dd-cf04-4f35-bb82-fce7996270a4/node-ca/0.log" Nov 02 21:08:32.266569 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:32.266549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:32.266654 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:32.266550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:32.266702 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:32.266664 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:32.266746 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:32.266553 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:32.266795 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:32.266767 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:32.266795 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:32.266704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:34.266897 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:34.266858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:34.266897 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:34.266886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:34.267396 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:34.266981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:34.267396 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:34.267002 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:34.267396 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:34.267070 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:34.267396 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:34.267138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:36.266797 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:36.266769 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:36.267189 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:36.266800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:36.267189 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:36.266861 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:36.267189 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:36.266884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:36.267189 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:36.266981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:36.267189 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:36.267024 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:38.267158 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:38.267128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:38.267158 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:38.267147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:38.267541 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:38.267162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:38.267541 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:38.267243 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:38.267541 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:38.267304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:38.267541 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:38.267375 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:40.266560 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:40.266530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:40.266901 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:40.266570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:40.266901 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:40.266627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:40.266901 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:40.266643 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:40.266901 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:40.266702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:40.266901 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:40.266763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:42.177993 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.177964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:42.178402 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.178009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:42.178402 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.178095 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:42.178402 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.178096 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:42.178402 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.178140 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs podName:bfa5c3f8-0ffe-4e56-adb0-ad12287596c6 nodeName:}" failed. No retries permitted until 2026-11-02 21:09:14.178127561 +0000 UTC m=+65.464418742 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs") pod "network-metrics-daemon-xk8mq" (UID: "bfa5c3f8-0ffe-4e56-adb0-ad12287596c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:42.178402 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.178152 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret podName:fddf083e-247f-4cdc-9722-3e6c2b43dd50 nodeName:}" failed. No retries permitted until 2026-11-02 21:09:14.178146274 +0000 UTC m=+65.464437455 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret") pod "global-pull-secret-syncer-xm28p" (UID: "fddf083e-247f-4cdc-9722-3e6c2b43dd50") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:42.266753 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.266734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:42.266753 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.266752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:42.266883 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.266808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:42.266915 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.266878 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:42.266946 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.266921 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:42.267019 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.267000 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:42.278269 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:42.278252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:42.278359 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.278349 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:42.278392 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.278361 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:42.278392 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.278371 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8djpn for pod openshift-network-diagnostics/network-check-target-pt5zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:42.278452 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:42.278402 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn podName:cbda9bc1-119f-4474-9cca-9204e084f4b6 nodeName:}" failed. No retries permitted until 2026-11-02 21:09:14.278393376 +0000 UTC m=+65.564684562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8djpn" (UniqueName: "kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn") pod "network-check-target-pt5zs" (UID: "cbda9bc1-119f-4474-9cca-9204e084f4b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:44.267414 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:44.267385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:44.267839 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:44.267388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:44.267839 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:44.267486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:44.267839 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:44.267388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:44.267839 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:44.267556 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:44.267839 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:44.267607 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:46.266619 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:46.266586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:46.266619 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:46.266611 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:46.267104 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:46.266669 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:46.267104 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:46.266595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:46.267104 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:46.266793 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:46.267104 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:46.266842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:48.267514 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.267482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:48.268103 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.267527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:48.268390 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.268349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:48.268390 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.268142 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:48.268390 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.267559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:48.268652 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.268458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:48.670078 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.669836 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pt5zs"] Nov 02 21:08:48.670217 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.670161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:48.670288 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.670264 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:48.674024 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.673999 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xm28p"] Nov 02 21:08:48.674153 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.674089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:48.674215 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.674178 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:48.674507 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.674478 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk8mq"] Nov 02 21:08:48.674588 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:48.674561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:48.674644 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:48.674633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:50.267262 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:50.267229 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:50.267675 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:50.267283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:50.267675 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:50.267283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:50.267675 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:50.267361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:50.267675 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:50.267423 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:50.267675 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:50.267563 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:52.267575 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:52.267550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:52.268082 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:52.267550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:52.268082 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:52.267650 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk8mq" podUID="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" Nov 02 21:08:52.268082 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:52.267746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pt5zs" podUID="cbda9bc1-119f-4474-9cca-9204e084f4b6" Nov 02 21:08:52.268082 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:52.267552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:52.268082 ip-10-0-137-1 kubenswrapper[2580]: E1102 21:08:52.267832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xm28p" podUID="fddf083e-247f-4cdc-9722-3e6c2b43dd50" Nov 02 21:08:54.015254 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.015181 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeReady" Nov 02 21:08:54.015757 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.015271 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:08:54.161234 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.161212 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tgsfm"] Nov 02 21:08:54.174266 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.174243 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hdm6g"] Nov 02 21:08:54.174369 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.174355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.184997 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.184974 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdm6g"] Nov 02 21:08:54.184997 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.184992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.185122 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.185023 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tgsfm"] Nov 02 21:08:54.208599 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.208577 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q7knd"] Nov 02 21:08:54.222878 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.222826 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:08:54.222989 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.222910 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:08:54.222989 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.222916 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\"" Nov 02 21:08:54.222989 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.222973 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:08:54.222989 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.222981 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:08:54.223124 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.223082 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.233350 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.233332 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:08:54.233350 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.233345 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:08:54.233472 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.233362 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:08:54.233472 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.233393 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\"" Nov 02 21:08:54.266936 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.266888 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7knd"] Nov 02 21:08:54.267048 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.267037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:08:54.267253 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.267230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:08:54.267488 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.267288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:08:54.267938 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.267918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.268115 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.268286 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.268286 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkgf\" (UniqueName: \"kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.268435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.268435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.268435 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.268390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f8w\" (UniqueName: \"kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.325496 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.325479 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:08:54.325956 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.325939 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:08:54.326035 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.325983 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:08:54.326075 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326061 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:08:54.326234 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326221 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\"" Nov 02 21:08:54.326301 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326289 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\"" Nov 02 21:08:54.326344 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326297 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:08:54.326344 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326311 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\"" Nov 02 21:08:54.326344 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326339 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\"" Nov 02 21:08:54.326438 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.326369 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:08:54.369587 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.369663 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.369663 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.369663 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f8w\" (UniqueName: \"kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.369773 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.369806 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpqr\" (UniqueName: \"kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.369848 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.369848 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.369942 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.370015 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.369977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.370015 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.370007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkgf\" (UniqueName: \"kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.370134 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.370115 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.370194 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.370162 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.370299 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.370281 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.373437 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.373412 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.373542 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.373527 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.422308 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.422279 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f8w\" (UniqueName: \"kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.422436 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.422420 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkgf\" (UniqueName: \"kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.470695 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.470670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.470785 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.470723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpqr\" (UniqueName: \"kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.470785 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.470751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.470785 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.470773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.471075 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.471059 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.471250 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.471234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.472395 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.472380 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.519101 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.519051 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:08:54.523116 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.523099 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpqr\" (UniqueName: \"kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.528017 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.528001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:08:54.555568 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.555545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:54.941618 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.941595 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hdm6g"] Nov 02 21:08:54.992313 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.992294 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tgsfm"] Nov 02 21:08:54.994509 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:54.994492 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7knd"] Nov 02 21:08:54.997965 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:54.997934 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8bc690_2b44_4fd3_a2d2_62da0962ec4e.slice/crio-0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b WatchSource:0}: Error finding container 0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b: Status 404 returned error can't find the container with id 0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b Nov 02 21:08:55.001201 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:55.001073 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd6e1a3_ff2f_4b77_97a7_33a39ab3eb97.slice/crio-67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b WatchSource:0}: Error finding container 67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b: Status 404 returned error can't find the container with id 67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b Nov 02 21:08:55.471412 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:55.471385 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7knd" event={"ID":"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97","Type":"ContainerStarted","Data":"67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b"} Nov 02 21:08:55.472827 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:55.472806 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgsfm" event={"ID":"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e","Type":"ContainerStarted","Data":"415353b9cfb6f6a55b14c266dbf3edf49a883cc4f40d6e8ba8bf72290a72d147"} Nov 02 21:08:55.472931 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:55.472853 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgsfm" event={"ID":"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e","Type":"ContainerStarted","Data":"0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b"} Nov 02 21:08:55.473767 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:55.473746 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdm6g" event={"ID":"d0f84692-9d39-43f4-ac6a-c78d837e0107","Type":"ContainerStarted","Data":"eff3d924ba85cff135d93224bcd3c5faade1cd181a078c47f42751344c48b572"} Nov 02 21:08:56.478328 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:56.478294 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgsfm" event={"ID":"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e","Type":"ContainerStarted","Data":"b21c0ebb24b92bc99444efb3d5586189d64eb93de3ccc97d8fb2ed46f43d45db"} Nov 02 21:08:57.055588 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.055527 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69"] Nov 02 21:08:57.058825 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.058805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.102164 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.102140 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2"] Nov 02 21:08:57.105182 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.105164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.119398 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.119377 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nwd24\"" Nov 02 21:08:57.119511 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.119444 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:08:57.148063 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.148036 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 21:08:57.148154 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.148064 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:08:57.148154 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.148106 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:08:57.154470 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.154455 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:08:57.155245 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.155221 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69"] Nov 02 21:08:57.166495 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.166474 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6phbr"] Nov 02 21:08:57.169340 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.169321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.173825 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.173804 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2"] Nov 02 21:08:57.190489 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.190573 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.190573 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.190573 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.190679 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.190679 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.190745 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.190745 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk29p\" (UniqueName: \"kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.190812 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfkf\" (UniqueName: \"kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.190843 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.190834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.208077 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.208050 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:08:57.208373 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.208333 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bhgh7\"" Nov 02 21:08:57.208373 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.208362 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 21:08:57.208481 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.208468 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 21:08:57.291107 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.291192 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.291192 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291192 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.291192 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.291377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk29p\" (UniqueName: \"kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.291377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfkf\" (UniqueName: \"kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.291377 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291581 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96bhz\" (UniqueName: \"kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291581 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291581 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.291707 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.291707 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.291707 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291707 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291707 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.291930 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.291730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.292455 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.292109 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.292455 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.292413 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.292580 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.292474 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.292631 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.292617 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.295144 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.295123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.295144 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.295136 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.295292 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.295131 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.295483 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.295465 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.308686 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.308637 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:08:57.308686 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.308637 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:08:57.308825 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.308703 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b7xvl\"" Nov 02 21:08:57.308825 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.308730 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:08:57.345694 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.345652 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfkf\" (UniqueName: \"kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.346047 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.346021 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk29p\" (UniqueName: \"kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.374237 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.374209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:08:57.392390 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392504 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392504 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392504 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392504 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392501 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392533 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392698 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.392934 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.392733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96bhz\" (UniqueName: \"kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.394537 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.393697 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.394537 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.394133 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.394537 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.394475 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.398928 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.398907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.399081 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.398991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.421016 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.420980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:08:57.447245 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.447221 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96bhz\" (UniqueName: \"kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.492052 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.485007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:08:57.496335 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:57.496306 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93eb21de_12ad_49fb_b162_b53f3763d2ce.slice/crio-e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8 WatchSource:0}: Error finding container e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8: Status 404 returned error can't find the container with id e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8 Nov 02 21:08:57.786047 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.786010 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69"] Nov 02 21:08:57.834177 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:57.834150 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992df832_d24f_4517_ab42_d47ff9f51b7e.slice/crio-6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd WatchSource:0}: Error finding container 6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd: Status 404 returned error can't find the container with id 6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd Nov 02 21:08:57.840131 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:57.840107 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2"] Nov 02 21:08:57.848594 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:08:57.848567 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc812be04_0eac_4a4c_9456_60332f6fa93d.slice/crio-e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680 WatchSource:0}: Error finding container e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680: Status 404 returned error can't find the container with id e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680 Nov 02 21:08:58.486052 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:58.485986 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" event={"ID":"c812be04-0eac-4a4c-9456-60332f6fa93d","Type":"ContainerStarted","Data":"e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680"} Nov 02 21:08:58.488392 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:58.488362 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" event={"ID":"992df832-d24f-4517-ab42-d47ff9f51b7e","Type":"ContainerStarted","Data":"971b98165f775159b19fc156696229cd2105b2e6158a55de5d42bb7dccbf2c36"} Nov 02 21:08:58.488541 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:58.488417 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" event={"ID":"992df832-d24f-4517-ab42-d47ff9f51b7e","Type":"ContainerStarted","Data":"a6a5df886c34b22f7f86ed072117803b1aead8fa057c33c9ebf1a69fae0e1c6a"} Nov 02 21:08:58.488541 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:58.488436 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" event={"ID":"992df832-d24f-4517-ab42-d47ff9f51b7e","Type":"ContainerStarted","Data":"6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd"} Nov 02 21:08:58.489712 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:58.489669 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6phbr" event={"ID":"93eb21de-12ad-49fb-b162-b53f3763d2ce","Type":"ContainerStarted","Data":"e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8"} Nov 02 21:08:59.354396 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.354341 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:08:59.357844 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.357829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.406945 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.407050 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.407103 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.407119 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.407200 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:08:59.407270 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.407233 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbphx\"" Nov 02 21:08:59.473929 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.473897 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:08:59.474060 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.473988 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:08:59.474116 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.474063 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:08:59.494970 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.494474 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7knd" event={"ID":"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97","Type":"ContainerStarted","Data":"a63a32ac59feaa3f337a9f9f0c0d783967dccfba74fd291eef63d711cddf2a65"} Nov 02 21:08:59.494970 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.494569 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q7knd" Nov 02 21:08:59.494970 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.494592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7knd" event={"ID":"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97","Type":"ContainerStarted","Data":"d46b48c5d8eb31b8e9fb6216050561b0b05f173848cc27904a1d78f488569588"} Nov 02 21:08:59.513556 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513680 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513680 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513783 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513783 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513883 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513883 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513993 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.513993 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.513975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.514102 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.514001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.514102 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.514031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.514102 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.514068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.514242 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.514134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.564593 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.564570 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:08:59.566618 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.566598 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:08:59.615488 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615596 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615596 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615596 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615596 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615596 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.615838 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.615800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.617097 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.617059 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.617358 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.617333 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.617898 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.617872 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.619035 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.619013 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.619645 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.619616 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.619784 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.619745 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.620225 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.620200 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.620378 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.620348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.620697 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.620666 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.621025 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.621002 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.621244 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.621221 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.621348 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.621332 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.623476 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.623439 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-q7knd" podStartSLOduration=1.549309231 podStartE2EDuration="5.623428061s" totalImagesPullingTime="4.07411883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:59.622007281 +0000 UTC m=+50.908298485" watchObservedRunningTime="2026-11-02 21:08:59.623428061 +0000 UTC m=+50.909719263" Nov 02 21:08:59.670094 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.670069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.673348 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.673326 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d55f57bf6-46r5k"] Nov 02 21:08:59.678347 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.678331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.685359 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.685343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:08:59.727726 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.727701 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 21:08:59.728014 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.727978 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7dlcpn653nlg\"" Nov 02 21:08:59.728116 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.728072 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dzqlm\"" Nov 02 21:08:59.728185 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.728151 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 21:08:59.728244 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.728233 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 21:08:59.728296 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.728271 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 21:08:59.728346 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.728304 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 21:08:59.735227 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.734945 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d55f57bf6-46r5k"] Nov 02 21:08:59.817530 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817530 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817742 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817742 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817742 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817889 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817889 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xssrx\" (UniqueName: \"kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.817889 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.818048 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.817924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919078 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919186 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssrx\" (UniqueName: \"kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919186 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919186 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919336 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919336 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919336 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919336 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.919336 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.919312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.920261 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.920237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.921788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.921277 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.923077 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.923045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.923350 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.923330 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.923426 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.923332 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.923726 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.923707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.925017 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.924896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.925017 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.924977 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:08:59.972152 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:08:59.972126 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssrx\" (UniqueName: \"kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:00.004641 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.004615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:00.109906 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.109874 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:00.116522 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:00.116491 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c65bcf_b0a3_4109_8b9b_21e227f8b9e7.slice/crio-3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88 WatchSource:0}: Error finding container 3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88: Status 404 returned error can't find the container with id 3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88 Nov 02 21:09:00.385739 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.385713 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d55f57bf6-46r5k"] Nov 02 21:09:00.393992 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:00.393964 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e77bc5a_0603_4b53_900d_c3335549d101.slice/crio-7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c WatchSource:0}: Error finding container 7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c: Status 404 returned error can't find the container with id 7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c Nov 02 21:09:00.506781 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.506667 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88"} Nov 02 21:09:00.509184 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.509067 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgsfm" event={"ID":"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e","Type":"ContainerStarted","Data":"f43772bd14ebef6bf4d9c293e6acc0a0fc538762fd271f58274bea430c72b16e"} Nov 02 21:09:00.511288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.511268 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" event={"ID":"c812be04-0eac-4a4c-9456-60332f6fa93d","Type":"ContainerStarted","Data":"502f7af30432a9d0c3df1c94fa9f8e2a3753d9a64221a2ee16a13b0d9cc88c7d"} Nov 02 21:09:00.511364 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.511320 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" event={"ID":"c812be04-0eac-4a4c-9456-60332f6fa93d","Type":"ContainerStarted","Data":"ccc1929d88ecf5136ccadff4d0fae1b5be4c35d9997ca5a5fd599723c4ca0479"} Nov 02 21:09:00.512617 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.512599 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c"} Nov 02 21:09:00.619595 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:00.619495 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tgsfm" podStartSLOduration=1.892674513 podStartE2EDuration="6.619475799s" totalImagesPullingTime="4.726801286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:00.618005011 +0000 UTC m=+51.904296213" watchObservedRunningTime="2026-11-02 21:09:00.619475799 +0000 UTC m=+51.905767003" Nov 02 21:09:01.518796 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.518760 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" event={"ID":"992df832-d24f-4517-ab42-d47ff9f51b7e","Type":"ContainerStarted","Data":"68447c6a6df7b9f68093ca22547eaf4ee67a39c2765f0fb8f00ded18fc8b721d"} Nov 02 21:09:01.524568 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.524532 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" event={"ID":"c812be04-0eac-4a4c-9456-60332f6fa93d","Type":"ContainerStarted","Data":"24841065116eb73dd483cc0bc8311c3b433fa6ae910497926848d4fe0b8c425b"} Nov 02 21:09:01.629805 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.629748 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" podStartSLOduration=1.91336398 podStartE2EDuration="4.629731121s" totalImagesPullingTime="2.716367141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:01.628254649 +0000 UTC m=+52.914545852" watchObservedRunningTime="2026-11-02 21:09:01.629731121 +0000 UTC m=+52.916022324" Nov 02 21:09:01.736908 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.736855 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" podStartSLOduration=2.2236499849999998 podStartE2EDuration="4.736837548s" totalImagesPullingTime="2.513187563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:01.735943904 +0000 UTC m=+53.022235105" watchObservedRunningTime="2026-11-02 21:09:01.736837548 +0000 UTC m=+53.023128751" Nov 02 21:09:01.875064 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.875024 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99xw5"] Nov 02 21:09:01.878459 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.878437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:01.927572 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.927540 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 21:09:01.927760 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.927655 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lnmnx\"" Nov 02 21:09:01.929253 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.929227 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b797d4856-8xgjr"] Nov 02 21:09:01.932741 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.932716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:01.933602 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.933585 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99xw5"] Nov 02 21:09:01.983730 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.983687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 21:09:01.983964 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.983934 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 21:09:01.991655 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:01.991625 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b797d4856-8xgjr"] Nov 02 21:09:02.027892 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.027844 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 21:09:02.028094 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.028080 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jxzwg\"" Nov 02 21:09:02.028153 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.028088 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 21:09:02.028153 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.028108 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2gs96mbktstv5\"" Nov 02 21:09:02.038274 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjgd\" (UniqueName: \"kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.038274 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038236 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038419 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038478 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038478 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.038540 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038598 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.038629 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.038619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.138979 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.138936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.138979 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.138993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139224 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139224 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.139224 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139224 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139224 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139453 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjgd\" (UniqueName: \"kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.139453 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139655 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139631 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.139837 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.139809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.140331 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.140305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.142081 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.142055 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.142183 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.142098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.142237 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.142220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.145440 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.145416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.193022 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.192983 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjgd\" (UniqueName: \"kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.193393 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.193368 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.246067 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.245831 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:02.492395 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.492356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:02.530183 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.530091 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hdm6g" event={"ID":"d0f84692-9d39-43f4-ac6a-c78d837e0107","Type":"ContainerStarted","Data":"a6caa50775614681f6ed8b606988859b2db13b61e860ae17181fbcb228e23504"} Nov 02 21:09:02.636722 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.636674 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hdm6g" podStartSLOduration=1.8337168849999999 podStartE2EDuration="8.636659172s" totalImagesPullingTime="6.802942287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:02.636464368 +0000 UTC m=+53.922755574" watchObservedRunningTime="2026-11-02 21:09:02.636659172 +0000 UTC m=+53.922950375" Nov 02 21:09:02.667071 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.667044 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b797d4856-8xgjr"] Nov 02 21:09:02.695253 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:02.695219 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9bc7cc_f7b4_45c8_a099_ba0d03a1b2c1.slice/crio-026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264 WatchSource:0}: Error finding container 026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264: Status 404 returned error can't find the container with id 026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264 Nov 02 21:09:02.882365 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:02.882333 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99xw5"] Nov 02 21:09:02.888525 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:02.888501 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0423cd6d_c791_4383_92cd_8b699aca0c0e.slice/crio-f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0 WatchSource:0}: Error finding container f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0: Status 404 returned error can't find the container with id f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0 Nov 02 21:09:03.439718 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.439693 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:03.534233 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.534204 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" event={"ID":"0423cd6d-c791-4383-92cd-8b699aca0c0e","Type":"ContainerStarted","Data":"f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0"} Nov 02 21:09:03.535352 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.535326 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" event={"ID":"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1","Type":"ContainerStarted","Data":"026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264"} Nov 02 21:09:03.536895 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.536874 2580 generic.go:356] "Generic (PLEG): container finished" podID="93eb21de-12ad-49fb-b162-b53f3763d2ce" containerID="f7295ed6a54d35a10bc5d08203238436d96f3ac4f6f7a85ccb6379adb94eedcd" exitCode=0 Nov 02 21:09:03.537021 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.536943 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6phbr" event={"ID":"93eb21de-12ad-49fb-b162-b53f3763d2ce","Type":"ContainerDied","Data":"f7295ed6a54d35a10bc5d08203238436d96f3ac4f6f7a85ccb6379adb94eedcd"} Nov 02 21:09:03.538488 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.538462 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="431ffc7f069c226092e6138a9629410989a912c6271afedbfa5c9ac2acc46c76" exitCode=0 Nov 02 21:09:03.538588 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:03.538544 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"431ffc7f069c226092e6138a9629410989a912c6271afedbfa5c9ac2acc46c76"} Nov 02 21:09:04.543733 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:04.543695 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6phbr" event={"ID":"93eb21de-12ad-49fb-b162-b53f3763d2ce","Type":"ContainerStarted","Data":"022b496a5002fb0940a2a67617da7b53e46815fee77d5a4fe2528230503e2493"} Nov 02 21:09:04.544198 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:04.543753 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6phbr" event={"ID":"93eb21de-12ad-49fb-b162-b53f3763d2ce","Type":"ContainerStarted","Data":"d13b08ec5bdeab6713dede52c66820f066dc83249976db840cd3d398f7a8c2ba"} Nov 02 21:09:04.652218 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:04.652175 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6phbr" podStartSLOduration=2.459672897 podStartE2EDuration="7.652162444s" totalImagesPullingTime="5.128203449s" totalInitContainerRuntime="64.286098ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:04.652070057 +0000 UTC m=+55.938361261" watchObservedRunningTime="2026-11-02 21:09:04.652162444 +0000 UTC m=+55.938453648" Nov 02 21:09:05.995711 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:05.995638 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:06.000538 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.000518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.096565 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096541 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:09:06.096664 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096589 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:09:06.096664 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096632 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:09:06.096788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096664 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:09:06.096788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096705 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:09:06.096788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096748 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:09:06.096788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.096782 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sb2rf\"" Nov 02 21:09:06.098364 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.098343 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:06.128604 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.128581 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:09:06.128686 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.128664 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:09:06.128686 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.128669 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:09:06.142863 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.142843 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ptqqmo5n63c4\"" Nov 02 21:09:06.163635 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.163615 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:09:06.163709 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.163687 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:09:06.163709 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.163699 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:09:06.177462 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177560 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177560 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177623 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177701 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177729 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177756 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177784 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177825 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177897 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.177974 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.177927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178031 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178072 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178072 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178163 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178208 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178253 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178288 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.178326 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.178300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.240502 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.240452 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:09:06.254090 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.254068 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:09:06.279404 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279471 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279471 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279471 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279625 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279625 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279705 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279705 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279705 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279818 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.279831 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.279984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.280020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280126 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.280061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280414 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.280393 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280490 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.280467 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.280712 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.280630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.281152 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.281127 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.282204 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.282179 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.282589 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.282568 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.282745 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.282699 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.283166 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.283142 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.283439 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.283400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.283965 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.283935 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.284368 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.284349 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.284450 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.284354 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.284549 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.284527 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.284850 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.284830 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.284919 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.284898 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.285212 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.285194 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.285403 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.285385 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.286744 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.286728 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.331930 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.331909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.339775 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.339760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:06.559623 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.559544 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"f78f97b0dee44eb0f2e33f3046bdf450eee380de8d5557bcb4cee93552d403c6"} Nov 02 21:09:06.559623 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.559610 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"91544da80afd0f015f91819d6e5cad88e7b6df9088e4fc5b70590689e621da6f"} Nov 02 21:09:06.559788 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.559635 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"08545046cf7a5fd4a73c169e2683c736413afb9653e3a5aa688a884438e0aa5f"} Nov 02 21:09:06.808386 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:06.808348 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:06.813918 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:06.813890 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823cc8d3_d22a_464b_b02a_d84d6b2e4cd8.slice/crio-c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad WatchSource:0}: Error finding container c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad: Status 404 returned error can't find the container with id c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad Nov 02 21:09:07.565960 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.565924 2580 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" exitCode=0 Nov 02 21:09:07.566332 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.566304 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} Nov 02 21:09:07.566434 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.566416 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad"} Nov 02 21:09:07.568254 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.567847 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" event={"ID":"0423cd6d-c791-4383-92cd-8b699aca0c0e","Type":"ContainerStarted","Data":"3839760dad44adbcaaea8effe5edd72cd937729fa38da451b461bb56847e6955"} Nov 02 21:09:07.568254 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.568068 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:07.573292 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.573273 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:07.786737 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:07.786652 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" podStartSLOduration=2.191190686 podStartE2EDuration="6.786637292s" totalImagesPullingTime="4.595446606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:07.786561418 +0000 UTC m=+59.072852618" watchObservedRunningTime="2026-11-02 21:09:07.786637292 +0000 UTC m=+59.072928495" Nov 02 21:09:08.573974 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.573940 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"0fc2b0df151abfef66ae99915bfd8cc3de5741099bbfde2b89ade866f671d529"} Nov 02 21:09:08.574289 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.574019 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"482a85adb6074ccd6166fa7920012e6cfd303d8347fd53bac0a2a549f43c6b9b"} Nov 02 21:09:08.574289 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.574046 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" event={"ID":"1e77bc5a-0603-4b53-900d-c3335549d101","Type":"ContainerStarted","Data":"a8f93c08ede9300148ca55350e6cd4f9cb5f9edd01212f0c409b60e5a1d05929"} Nov 02 21:09:08.574289 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.574098 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:08.575482 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.575463 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" event={"ID":"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1","Type":"ContainerStarted","Data":"ecde8756b95d95e50468c9fbaa48b1e85893feb711f4db4f29bccd35f6e6bc9a"} Nov 02 21:09:08.686439 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.686395 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" podStartSLOduration=1.8993349529999999 podStartE2EDuration="9.686381321s" totalImagesPullingTime="7.787046368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:08.684999136 +0000 UTC m=+59.971290340" watchObservedRunningTime="2026-11-02 21:09:08.686381321 +0000 UTC m=+59.972672523" Nov 02 21:09:08.795436 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:08.795378 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" podStartSLOduration=2.345386233 podStartE2EDuration="7.795362819s" totalImagesPullingTime="5.449976586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:08.794829896 +0000 UTC m=+60.081121099" watchObservedRunningTime="2026-11-02 21:09:08.795362819 +0000 UTC m=+60.081654022" Nov 02 21:09:09.514539 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.514504 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:09.582963 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.582913 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"1024676d51ec5ed833b1571d6dd5b2f4722fda9130472d355f19eefae878b686"} Nov 02 21:09:09.583381 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.583001 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"6bd2d593491bd89d6bfa0629d5300f76f8caaab635781d5e66c9f42daa5c05cf"} Nov 02 21:09:09.583381 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.583033 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"d58b4613479d46e8ade69d274ec5709efc830ebd5e355c48fa6b92fabe3d96e9"} Nov 02 21:09:09.583381 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.583061 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"5d7676560b535c3574a37c847bad94e8b8571b5826fb8ae875279a721531a277"} Nov 02 21:09:09.583381 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.583093 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"bdfb90f0780c2a7956f95f9decc4c8ab6ad529ddf53e57119c96f092f486dce6"} Nov 02 21:09:09.583381 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.583120 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerStarted","Data":"39ab7919b721f6f667a78fea1f155f20c9af38b93279fa0c0e9e516a9b04be44"} Nov 02 21:09:09.738509 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:09.738466 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.305576795 podStartE2EDuration="10.738452749s" totalImagesPullingTime="7.422469079s" totalInitContainerRuntime="10.406875ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:09.737512257 +0000 UTC m=+61.023803461" watchObservedRunningTime="2026-11-02 21:09:09.738452749 +0000 UTC m=+61.024743951" Nov 02 21:09:14.248513 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.248481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:14.248896 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.248529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:14.296670 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.296644 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:09:14.301781 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.301755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:14.343046 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.343024 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:09:14.349284 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.349260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:14.350861 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.350845 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:14.397171 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.397146 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:09:14.438068 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.438044 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\"" Nov 02 21:09:14.440086 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.440066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:14.447095 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.447073 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:09:14.452284 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.452263 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:14.452414 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.452398 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\"" Nov 02 21:09:14.455466 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.455445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:14.591351 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.591329 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:14.752052 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.752028 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\"" Nov 02 21:09:14.756157 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.756133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:14.825371 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.825346 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk8mq"] Nov 02 21:09:14.834167 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:14.834135 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa5c3f8_0ffe_4e56_adb0_ad12287596c6.slice/crio-13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c WatchSource:0}: Error finding container 13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c: Status 404 returned error can't find the container with id 13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c Nov 02 21:09:14.870432 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:14.870412 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xm28p"] Nov 02 21:09:14.875947 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:14.875927 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddf083e_247f_4cdc_9722_3e6c2b43dd50.slice/crio-4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3 WatchSource:0}: Error finding container 4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3: Status 404 returned error can't find the container with id 4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3 Nov 02 21:09:15.122857 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:15.122828 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pt5zs"] Nov 02 21:09:15.128443 ip-10-0-137-1 kubenswrapper[2580]: W1102 21:09:15.128421 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbda9bc1_119f_4474_9cca_9204e084f4b6.slice/crio-a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263 WatchSource:0}: Error finding container a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263: Status 404 returned error can't find the container with id a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263 Nov 02 21:09:15.605378 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:15.605335 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pt5zs" event={"ID":"cbda9bc1-119f-4474-9cca-9204e084f4b6","Type":"ContainerStarted","Data":"a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263"} Nov 02 21:09:15.606628 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:15.606601 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xm28p" event={"ID":"fddf083e-247f-4cdc-9722-3e6c2b43dd50","Type":"ContainerStarted","Data":"4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3"} Nov 02 21:09:15.607888 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:15.607859 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk8mq" event={"ID":"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6","Type":"ContainerStarted","Data":"13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c"} Nov 02 21:09:16.616513 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:16.616228 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk8mq" event={"ID":"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6","Type":"ContainerStarted","Data":"7d105c3ada6c3244fa55bb1af5e863a7b302aa388e000f4a8695def7222d9d2d"} Nov 02 21:09:16.616513 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:16.616291 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk8mq" event={"ID":"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6","Type":"ContainerStarted","Data":"adfadb9cf484d8e50f5bd46078c5ea2423040fa0279bad48979cd8b87f6b7eee"} Nov 02 21:09:16.620374 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:16.620216 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} Nov 02 21:09:16.620374 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:16.620319 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} Nov 02 21:09:16.723027 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:16.722979 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xk8mq" podStartSLOduration=66.190091395 podStartE2EDuration="1m7.722935549s" totalImagesPullingTime="1.532844154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:16.722587418 +0000 UTC m=+68.008878621" watchObservedRunningTime="2026-11-02 21:09:16.722935549 +0000 UTC m=+68.009226802" Nov 02 21:09:17.630341 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:17.630295 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} Nov 02 21:09:17.630815 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:17.630372 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} Nov 02 21:09:17.630815 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:17.630413 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} Nov 02 21:09:17.630815 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:17.630452 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerStarted","Data":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} Nov 02 21:09:17.751393 ip-10-0-137-1 kubenswrapper[2580]: I1102 21:09:17.751328 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.929692735 podStartE2EDuration="12.751309759s" totalImagesPullingTime="8.805360864s" totalInitContainerRuntime="16.25616ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:17.750732978 +0000 UTC m=+69.037024192" watchObservedRunningTime="2026-11-02 21:09:17.751309759 +0000 UTC m=+69.037600962" Nov 02 21:09:20.373312 ip-10-0-137-1 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:09:20.389683 ip-10-0-137-1 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:09:20.389905 ip-10-0-137-1 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:09:20.390099 ip-10-0-137-1 systemd[1]: kubelet.service: Consumed 3.990s CPU time. Nov 02 21:09:20.405415 ip-10-0-137-1 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:09:20.480889 ip-10-0-137-1 kubenswrapper[5859]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:09:20.480889 ip-10-0-137-1 kubenswrapper[5859]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:09:20.480889 ip-10-0-137-1 kubenswrapper[5859]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:09:20.480889 ip-10-0-137-1 kubenswrapper[5859]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:09:20.482812 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482742 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.482812 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482811 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.482812 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482815 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482819 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482821 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482824 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482828 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482831 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482834 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482836 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482839 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482842 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482845 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482849 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482853 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482857 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482860 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482863 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482866 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482869 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482871 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.482921 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482874 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482876 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482879 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482886 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482889 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482892 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482895 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482897 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482900 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482902 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482905 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482908 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482910 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482913 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482916 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482918 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482921 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482924 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482926 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482929 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.483372 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482932 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482935 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482938 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482940 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482943 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482946 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482964 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482967 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482970 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482973 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482975 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482978 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482980 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482983 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482986 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482989 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482991 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482994 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482996 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.483816 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.482999 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483001 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483004 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483006 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483009 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483015 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483018 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483020 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483024 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483026 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483028 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483031 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483035 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483038 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483041 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483043 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483046 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483048 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483051 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.484324 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483053 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483056 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483058 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483061 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483064 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483066 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483069 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483071 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483074 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483076 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483079 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483081 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483084 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483086 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483089 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483091 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483094 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483096 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483100 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483102 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.484769 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483106 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483110 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483114 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483117 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483121 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483124 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483127 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483852 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483859 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483863 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483866 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483869 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483871 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483874 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483877 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483886 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483889 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483891 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483894 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483897 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.485238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483899 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483902 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483904 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483907 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483909 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483912 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483916 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483920 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483923 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483925 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483930 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483933 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483936 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483939 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483942 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483944 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483963 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483966 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483969 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.485726 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483978 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483981 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483983 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483986 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483988 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483991 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483993 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483996 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.483999 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484001 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484004 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484007 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484009 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484012 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484014 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484017 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484020 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484023 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484025 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484028 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.486198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484030 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484033 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484035 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484040 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484044 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484047 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484050 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484052 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484061 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484064 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484066 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484069 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484071 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484080 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484083 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484086 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484088 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484091 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484093 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.486660 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484096 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484098 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484101 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484103 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484106 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484108 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484111 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484113 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484116 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484119 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484121 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484124 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484126 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484128 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484131 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484133 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484137 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484140 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484143 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484145 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.487114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484148 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484150 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484153 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484155 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484158 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484160 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484164 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484172 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484174 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484177 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484179 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484182 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484184 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484187 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484189 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484644 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484649 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484652 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484655 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.487593 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484658 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484660 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484663 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484665 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484668 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484670 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484672 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484675 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484677 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484683 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484687 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484690 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484693 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484695 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484698 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484700 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484703 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484705 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484708 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.488079 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484710 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484713 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484716 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484718 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484721 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484723 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484726 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484728 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484731 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484733 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484736 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484738 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484741 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484743 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484746 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484748 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484751 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484754 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484756 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484759 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.488556 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484761 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484764 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484768 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484771 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484774 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484777 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484780 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484782 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484785 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484788 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484790 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484793 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484796 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484798 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484801 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484804 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484806 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484809 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484811 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484814 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.489030 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484816 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484818 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484821 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484824 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484826 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484829 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484831 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484834 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484836 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484839 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484841 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484844 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484846 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484849 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484852 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484855 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484857 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484860 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484862 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484865 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.489566 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484867 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484870 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484872 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484875 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484877 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484879 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484882 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484885 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484887 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484890 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484892 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484896 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484899 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484902 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484904 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484907 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484909 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484912 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484915 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.490061 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484917 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484919 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484922 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.484925 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485032 5859 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485040 5859 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485046 5859 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485050 5859 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485055 5859 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485058 5859 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485062 5859 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485066 5859 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485069 5859 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485072 5859 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485075 5859 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485078 5859 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485082 5859 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485084 5859 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485087 5859 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485090 5859 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485093 5859 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485096 5859 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485101 5859 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:09:20.490502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485103 5859 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485107 5859 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485110 5859 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485113 5859 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485117 5859 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485120 5859 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485123 5859 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485127 5859 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485130 5859 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485132 5859 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485135 5859 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485138 5859 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485141 5859 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485145 5859 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485148 5859 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485151 5859 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485154 5859 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485159 5859 flags.go:64] FLAG: --enable-server="true" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485161 5859 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485165 5859 flags.go:64] FLAG: --event-burst="100" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485168 5859 flags.go:64] FLAG: --event-qps="50" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485171 5859 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485174 5859 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485177 5859 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:09:20.491711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485181 5859 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485184 5859 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485187 5859 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485190 5859 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485193 5859 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485196 5859 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485199 5859 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485202 5859 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485205 5859 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485208 5859 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485211 5859 flags.go:64] FLAG: --feature-gates="" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485217 5859 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485220 5859 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485223 5859 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485226 5859 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485229 5859 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485232 5859 flags.go:64] FLAG: --help="false" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485235 5859 flags.go:64] FLAG: --hostname-override="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485238 5859 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485241 5859 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485244 5859 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485249 5859 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485252 5859 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:09:20.492292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485255 5859 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485258 5859 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485265 5859 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485268 5859 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485271 5859 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485274 5859 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485277 5859 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485279 5859 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485282 5859 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485286 5859 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485288 5859 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485291 5859 flags.go:64] FLAG: --lock-file="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485294 5859 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485297 5859 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485300 5859 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485305 5859 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485308 5859 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485312 5859 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485314 5859 flags.go:64] FLAG: --logging-format="text" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485317 5859 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485321 5859 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485323 5859 flags.go:64] FLAG: --manifest-url="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485326 5859 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485330 5859 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485334 5859 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:09:20.492822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485337 5859 flags.go:64] FLAG: --max-pods="110" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485340 5859 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485343 5859 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485346 5859 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485349 5859 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485353 5859 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485357 5859 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485359 5859 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485366 5859 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485370 5859 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485374 5859 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485377 5859 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485380 5859 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485383 5859 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485385 5859 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485388 5859 flags.go:64] FLAG: --port="10250" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485391 5859 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485394 5859 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00bec3189046f4224" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485397 5859 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485400 5859 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485403 5859 flags.go:64] FLAG: --register-node="true" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485406 5859 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485410 5859 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485412 5859 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485415 5859 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485422 5859 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:09:20.493677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485426 5859 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485429 5859 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485432 5859 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485435 5859 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485438 5859 flags.go:64] FLAG: --runonce="false" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485441 5859 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485444 5859 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485447 5859 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485450 5859 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485452 5859 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485455 5859 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485460 5859 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485463 5859 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485466 5859 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485468 5859 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485472 5859 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485475 5859 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485478 5859 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485481 5859 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485484 5859 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485489 5859 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485492 5859 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485494 5859 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485499 5859 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485501 5859 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:09:20.494386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485504 5859 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485507 5859 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485510 5859 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485513 5859 flags.go:64] FLAG: --v="2" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485517 5859 flags.go:64] FLAG: --version="false" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485522 5859 flags.go:64] FLAG: --vmodule="" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485525 5859 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485529 5859 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485657 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485660 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485663 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485666 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485668 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485671 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485673 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485676 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485679 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485682 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485687 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485690 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485692 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.494942 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485695 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485698 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485701 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485704 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485706 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485709 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485711 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485714 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485716 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485719 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485721 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485724 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485726 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485729 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485731 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485734 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485737 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485740 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485742 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.495626 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485746 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485748 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485751 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485754 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485756 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485759 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485761 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485764 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485766 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485770 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485774 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485777 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485780 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485783 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485787 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485790 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485792 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485795 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485798 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485800 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.496253 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485802 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485805 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485808 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485810 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485813 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485815 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485818 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485820 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485823 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485826 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485828 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485831 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485833 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485837 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485839 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485841 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485844 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485846 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485849 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485851 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.496805 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485854 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485858 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485862 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485866 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485869 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485871 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485875 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485877 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485880 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485882 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485885 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485888 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485890 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485893 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485895 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485898 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485900 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485903 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485905 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.497312 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485908 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485911 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485913 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485916 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485918 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485921 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485923 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485927 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485929 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485932 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485934 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485937 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485939 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485942 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.485944 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.485983 5859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:09:20.497744 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486017 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486021 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486025 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486029 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486032 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486035 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486038 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486041 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486044 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486046 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486049 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486052 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486055 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486057 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486060 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486063 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486065 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486068 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486071 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486073 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.498303 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486076 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486078 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486080 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486083 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486086 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486089 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486091 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486094 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486096 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486099 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486101 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486104 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486108 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486110 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486113 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486117 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486119 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486122 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486124 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.498843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486127 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486129 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486132 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486135 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486139 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486142 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486145 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486148 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486151 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486153 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486156 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486159 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486161 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486164 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486166 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486169 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486171 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486174 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486177 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.499448 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486179 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486182 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486184 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486187 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486191 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486193 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486198 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486201 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486204 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486208 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486211 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486214 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486216 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486218 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486221 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486224 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486226 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486230 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486233 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486236 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.500220 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486238 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486241 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486243 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486246 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486248 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486251 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486254 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486256 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486258 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486261 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486263 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486266 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486269 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486271 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486274 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486276 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486279 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486281 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486285 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486288 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.501031 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486290 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486294 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486298 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486300 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486302 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486305 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486308 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.486310 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.489903 5859 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.489920 5859 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.489937 5859 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.489941 5859 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490007 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490012 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490015 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490018 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490021 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490024 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490028 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490030 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.501813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490033 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490036 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490038 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490041 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490044 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490046 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490049 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490052 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490054 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490057 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490060 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490063 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490065 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490068 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490071 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490074 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490076 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490079 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490081 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.502732 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490084 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490086 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490089 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490092 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490094 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490097 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490099 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490102 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490104 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490107 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490109 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490112 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490115 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490117 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490120 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490122 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490125 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490127 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490130 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490132 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.503509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490135 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490138 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490140 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490143 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490145 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490147 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490150 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490152 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490155 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490157 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490159 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490162 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490164 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490167 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490169 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490172 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490175 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490177 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490180 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.504331 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490182 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490185 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490187 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490190 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490192 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490195 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490198 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490201 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490205 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490208 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490211 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490214 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490216 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490219 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490221 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490225 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490229 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490232 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490235 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.505153 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490237 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490240 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490242 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490245 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490247 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490250 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490252 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490255 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490257 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490259 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490262 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490265 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490267 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490270 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490272 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490275 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490278 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490280 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490283 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490285 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.505910 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490287 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.490313 5859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490439 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490443 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490446 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490449 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490452 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490455 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490457 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490460 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490463 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490465 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490467 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490470 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490472 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490475 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.506759 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490478 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490480 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490482 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490485 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490487 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490490 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490492 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490495 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490498 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490500 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490503 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490505 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490508 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490511 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490513 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490516 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490519 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490521 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490523 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490526 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.507415 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490528 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490531 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490533 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490536 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490538 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490541 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490545 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490548 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490550 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490553 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490555 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490558 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490561 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490563 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490565 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490568 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490571 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490575 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490579 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.508241 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490582 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490585 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490587 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490590 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490592 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490595 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490598 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490601 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490603 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490606 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490608 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490611 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490613 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490616 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490619 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490621 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490625 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490629 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490632 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.509091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490636 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490639 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490641 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490644 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490646 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490649 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490652 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490655 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490657 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490660 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490662 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490665 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490667 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490670 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490672 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490675 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490678 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490680 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490682 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.509904 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490685 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490688 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490690 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490693 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490695 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490698 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490700 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490703 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490705 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490708 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490710 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490713 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490715 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490719 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490722 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.510798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.490744 5859 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490774 5859 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490778 5859 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490781 5859 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490783 5859 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490786 5859 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490789 5859 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490791 5859 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490794 5859 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490796 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490799 5859 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490802 5859 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490804 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490807 5859 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490810 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490812 5859 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490815 5859 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490818 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490820 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:09:20.511586 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490823 5859 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490825 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490828 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490830 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490833 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490835 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490838 5859 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490840 5859 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490843 5859 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490846 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490848 5859 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490851 5859 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490854 5859 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490858 5859 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490862 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490864 5859 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490867 5859 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490869 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490872 5859 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:09:20.512451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490875 5859 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490877 5859 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490879 5859 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490882 5859 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490884 5859 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490887 5859 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490889 5859 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490892 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490894 5859 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490897 5859 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490900 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490903 5859 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490905 5859 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490909 5859 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490912 5859 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490915 5859 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490918 5859 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490921 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490924 5859 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490927 5859 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:09:20.513257 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490930 5859 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490932 5859 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490935 5859 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490937 5859 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490940 5859 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490944 5859 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490965 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490968 5859 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490971 5859 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490974 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490977 5859 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490979 5859 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490982 5859 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490985 5859 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490987 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490990 5859 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490993 5859 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490995 5859 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.490998 5859 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:09:20.514130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491000 5859 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491003 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491005 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491008 5859 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491013 5859 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491016 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491018 5859 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491021 5859 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491023 5859 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491026 5859 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491028 5859 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491031 5859 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491033 5859 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491036 5859 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491038 5859 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491041 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491044 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491046 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491050 5859 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:09:20.514782 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491053 5859 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491056 5859 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491058 5859 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491061 5859 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491064 5859 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491066 5859 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491069 5859 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491071 5859 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491074 5859 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491076 5859 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:20.491079 5859 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.491124 5859 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.491988 5859 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492040 5859 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492226 5859 server.go:1014] "Starting client certificate rotation" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492365 5859 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492411 5859 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:03:09 +0000 UTC" deadline="2028-04-05 00:20:57.405087135 +0000 UTC" Nov 02 21:09:20.515310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492438 5859 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12459h11m36.912651186s" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.492637 5859 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.494824 5859 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.496086 5859 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.497969 5859 log.go:25] "Validated CRI v1 image API" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.498702 5859 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:09:20.516029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.501903 5859 fs.go:135] Filesystem UUIDs: map[5001333b-80af-4f93-9425-cce0ac96e88f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b023e6e7-bcf4-424e-8270-371d20d8db7c:/dev/nvme0n1p3] Nov 02 21:09:20.516395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.501928 5859 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3/userdata/shm major:0 minor:447 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263/userdata/shm major:0 minor:449 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eff3d924ba85cff135d93224bcd3c5faade1cd181a078c47f42751344c48b572/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eff3d924ba85cff135d93224bcd3c5faade1cd181a078c47f42751344c48b572/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/018ea93d-84d6-4e86-98ce-e376674b7a12/volumes/kubernetes.io~projected/kube-api-access-87gcx:{mountpoint:/var/lib/kubelet/pods/018ea93d-84d6-4e86-98ce-e376674b7a12/volumes/kubernetes.io~projected/kube-api-access-87gcx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~projected/kube-api-access-hxjgd:{mountpoint:/var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~projected/kube-api-access-hxjgd major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04d04d90-6021-4329-b47d-a41847969eef/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/04d04d90-6021-4329-b47d-a41847969eef/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~projected/kube-api-access-kfs9k:{mountpoint:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~projected/kube-api-access-kfs9k major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~projected/kube-api-access-xssrx:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~projected/kube-api-access-xssrx major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4238a8dd-cf04-4f35-bb82-fce7996270a4/volumes/kubernetes.io~projected/kube-api-access-x8jds:{mountpoint:/var/lib/kubelet/pods/4238a8dd-cf04-4f35-bb82-fce7996270a4/volumes/kubernetes.io~projected/kube-api-access-x8jds major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61d0c582-d441-4746-8619-d43d8f4099fe/volumes/kubernetes.io~projected/kube-api-access-wl2sd:{mountpoint:/var/lib/kubelet/pods/61d0c582-d441-4746-8619-d43d8f4099fe/volumes/kubernetes.io~projected/kube-api-access-wl2sd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~projected/kube-api-access-2nkgf:{mountpoint:/var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~projected/kube-api-access-2nkgf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~projected/kube-api-access-5zpqr:{mountpoint:/var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~projected/kube-api-access-5zpqr major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~projected/kube-api-access-phsrf:{mountpoint:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~projected/kube-api-access-phsrf major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/config-reloader/2 major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/prometheus/15 major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/kube-api-access-j8ws4:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/kube-api-access-j8ws4 major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/tls-assets major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/config major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/web-config major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~projected/kube-api-access-96bhz:{mountpoint:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~projected/kube-api-access-96bhz major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~projected/kube-api-access-ngfkf:{mountpoint:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~projected/kube-api-access-ngfkf major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/alertmanager/9 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/kube-api-access-sscd5:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/kube-api-access-sscd5 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/tls-assets major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/config-volume major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/web-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf62419-1dfd-4003-937c-6f3a835bd88e/volumes/kubernetes.io~projected/kube-api-access-vqqq6:{mountpoint:/var/lib/kubelet/pods/baf62419-1dfd-4003-937c-6f3a835bd88e/volumes/kubernetes.io~projected/kube-api-access-vqqq6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~projected/kube-api-access-n7fdm:{mountpoint:/var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~projected/kube-api-access-n7fdm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~projected/kube-api-access-sk29p:{mountpoint:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~projected/kube-api-access-sk29p major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbda9bc1-119f-4474-9cca-9204e084f4b6/volumes/kubernetes.io~projected/kube-api-access-8djpn:{mountpoint:/var/lib/kubelet/pods/cbda9bc1-119f-4474-9cca-9204e084f4b6/volumes/kubernetes.io~projected/kube-api-access-8djpn major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~projected/kube-api-access-75f8w:{mountpoint:/var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~projected/kube-api-access-75f8w major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~projected/kube-api-access-tk7q6:{mountpoint:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~projected/kube-api-access-tk7q6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efcdbaab-36bb-4f9a-9008-4e21d230deea/volumes/kubernetes.io~projected/kube-api-access-thd2l:{mountpoint:/var/lib/kubelet/pods/efcdbaab-36bb-4f9a-9008-4e21d230deea/volumes/kubernetes.io~projected/kube-api-access-thd2l major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbdffb38-5e8e-4484-a77a-abaea4c5aca4/volumes/kubernetes.io~projected/kube-api-access-8vzks:{mountpoint:/var/lib/kubelet/pods/fbdffb38-5e8e-4484-a77a-abaea4c5aca4/volumes/kubernetes.io~projected/kube-api-access-8vzks major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fddf083e-247f-4cdc-9722-3e6c2b43dd50/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fddf083e-247f-4cdc-9722-3e6c2b43dd50/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:443 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c1c73321845b056ac56f498d5c71bf249d6b05b6c402fd75585d9d986beaec94/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fcf78e9c4695b6b34fdfb67f604ec767559249056ecead4b62b133f6aa9c1f50/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/588703e854749d4051876e08e1c1ff5ec2086f5ab6ab9e7fa8eea9a9a44b708c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9e96cfb5034fefea6d233191aa72d0d1a10b9fee1c15982e538b9b4fefc37d16/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/37835218b545d35a867cf477d53989dce31838b52fc05e132724ce82983917f2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/05d5e11b62d84ab715f11df2ba9f89a1413d6b91b6fb24deb7c85a32bcc25c4c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/72baa5db4df889905f10b318688613664b792ed2d03f9fd0cde9c0f98bd753e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/12c000e05ff883abed8c07d0f99f3fb7723d5a54f8ec46f2eb37656626f08181/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f338a7e5262e5f411ea32b0ef1a9c5c2902063b3bf8b857034a3f18945384bf5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8ecd4c85166905443636b266843dcdfc7a2a0c302e6e38d0b811cd1e51637490/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c23896fb9130680855ca085c44174aeff0be806665d64e9f5583aa87a28e0ec0/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a0e7ac4dd322d968c1464231a01f07061814140f3d860c8489387e709ead4cfe/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f7f00b3470817b8fef32f08d917577cb8c9e2f01c2ea913364cadb2e16bb6c98/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/577b545a49ae7d24e8f332036575353816c1690852371115a4c1f3c56d91e899/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4a44c4df3427b6e34e9a54c06aaa6ebca1567cc4f6236c2be3fe7c8d3cc5d8ae/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/74f32a3e5b4831c99399e24151af1995433249f1a1dda5df762373608c824c41/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c0e744226647f08b1d22f212497ccfcbdfc066d2aedb94aff8f2107f18bb49ab/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e0cb5c2a7a1d2815b69dcc517e62b14487294667e35daaed1a7f0189d4c7aa8f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4ca6b64b1fe2ab2bd7ce4a841405f37df506684079cc823711f6e19cdecd0cca/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fd5a5038fcf5c17d9fe3234e3e72fb086c37ccdbfda1bd19b707b1218434b3cb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c2c9205077595d1cff025ffc9f24d6cd19daef36e550418a40f0c9db3d25a3d9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4abc39161093be2778879035b9f8295eeaa963b0d6cf8b344f8e176fee9d41d7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f696da219bf937f7b6b8bb45e16622d07b3a5d125a78a219587ef8e576450731/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/44f2ea4f5646cea8acae77fabe2a2d2f78c569171adc9f6b18715fc81eb91af4/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7f70087fb44c28371a4ac6e8ed4629cc7905934eaff058e77bbf4cb01a6c3727/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/6de520ceda7c44e107eb3bb30517751b39a7512c0790dda0bb97da7fa9811b3e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/7e0a5ae6d18866036c52506f6df23a9b7073d2d4ad896317105513e05c86f366/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/0b6ad0e5385166e2291a353ef22c9efabec8255b50da01ae17663044c8aa45f2/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2e0f73be6c6cf0e3a034a0c01cb1f58d1fe544bea949ab4eb8392dde3e76438d/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/65075c3e170e56102c69a4c575d93f00b19fab86684694d174624dfa7bcd2393/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/029266f5662d454a6be56e8b6eb88e8e1319376aa831d1fbee7b7cc81cbb0f7c/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e950f3e7094dcebbb8af011b43cbd6c4186bfd02f0cc8cff07fba9afddd227a9/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e5b94092253e4105e38dae5a62e9dc209864f85c8b81adbcea484153890f421c/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/d6b22e6a3a0ca78acc6e0ca480249168beee003430816c03938ce7c94e5465ea/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/57b66ff0a769a31a6194b494d7d97db364e100963795e93850bfb4c905d724de/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/c5d970f0d01a3e352b28f605b5ef24691b82b1aad5791f6e8669d79bbe711e30/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/e0cd8d5a07519650156d232c13dd9a9e16088283076567c06cdf1d985edc4b38/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/1eb234236fc22d86a4fec8f2ab7416ed37d9c6ebdfb571dd7e55f668a3c89c8d/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/223dcdb0d44447ad858cbe40fca7c9a82be4207ebcb4667b9c907af4b42c5a7f/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/771098b30a982db8da7317ce3be3dfdaf44779d935b8cd32518a7223308197ae/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/b92719cc98a11c7b34073fd7f184ade51a06f72ec51d6b2b34cc92e2ae0c864c/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/47a60b7b4fa3aef1969d92c482cc72c1cf0833f0585f435a01c4e412e9df47a8/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/360462f7b58b6f0d0c94f3fd095ba1062f72a5fffa6f0874fbd026500579be24/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/b411644a1e1da30b4a72d4b91aec5198bbf9a538efdb01649fc0cc5c2dc7940d/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/7cd968cf1976a0f6ee5e11d4b0b718aee8d141048839ec850a351585b44ffce3/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/64bfade55a07831dfea015012b09cb30a47bdbe633e2aca9668e22b1d2cf1d50/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/7f8e41a9cb7bf07a5f0291ad8eb77a78640cd0fc70225f25436b078eafb0ea0d/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/da929ed880ec2c688a327a09345e78b06034583cd7d0b3cca588293bb07f9bc5/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/76e4112c975e497a6054b0c9f29b92973ee39a2bce2714f4dcd35ca3e6b55cd8/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/62a2f4911240f2101fb4392a3ea42ec8931bb6eed46285bf9c85535114614b60/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/07c91e79d24f4ba22871effaec4d8a2b3975a0e0636c60084f3f22ddfaa4b054/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/4523300e80169746664f0d4c266502f408d5c38be6e2323876fccedace8c31f9/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/052fed5d51a6a889af9bb0559ba98df3af669a83af1c6130f5fc85058664b28b/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/7d8c3edee72097caab8aa707d3ecfd7048cf072489ca172e51bb19f729767d78/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/21056a4b38a630e9420ffa96953dc736f74583ccac6781c48c1ed7010e36c750/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/3567fea54efbbd5979b35a65bcec2d588a8c163914117bc4ace85feb3fa53bea/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/e61586a44725b686fd2f3de4aa4e89d641ac071b4c4971ed133013edc495c1b5/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/bbd3e68dda2ff5acbdf2c23477aaa59db59c455b3475eb08aa3e6e6a19e1c257/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/7a7255a7ef431432c4229f190da1928c4af601f2239ec5892447aa80dd4e55e0/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/75c53a44c7d73adf43b1db74c7c95d7cf5ba59e6148591390d7fc6c398325728/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/72ef1598a960b2ecb8804d51d97fbb3ab9b38f708a313fad22d9656d376277cb/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/96b2541c7a4246935b5ce0c6c71b425255be9a39c4160223f0d08aaa19ba718c/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/52c4d6b7420ca3c78ee2e389d68ca811edd53a65d19736d67293052391ab6b12/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/4b6789d90e2566a44f2245e7c11ba8ee47cb90fb11e4d69b031a7c25ca063c20/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/feff2ec78061bd6bc5962dc3f50f98ef1da029b6f9cde88e99221721a3910ac7/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/419d8ee730ede74d01ebd3e466aa155227635f0fba2fee11578f085b903341d0/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/bc5729ce7daecc5d50a60ed929a7d11fe958a33f93fb31ce282bebf881498956/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/16e3e5c7c03d4b4a6b81fedb5facc9193b46260515457a691bd7030aabf4919d/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/453992f05e49271cd9f007af3898a1f6c464975ba09f9650f935c84a0afeb014/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/8340fcbcb096d25baaa8baf4fa38d2c7a00c11d653808e2d1b114aa51f10f93e/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/de80a11e6d44a8319f817e3d92c4b2bbc1ee6fffb569011682b3bcd999f8cc5d/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/f65894c97a713a3934383ad612a19957a60e3193f28fefeb26854a35a32b1c44/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/93d1003117ec0c99e7fdefe887ac5e1733894a92bbc336d3851c00d4c5aea0b0/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/b45aff83c227f6cadadf7c5160bdba7b6c7c418d7e86eec5ddca93f992ad5094/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/3b4b2c77bb3f888220413218875f4f0cf9f7c500ac3708eb8fccf5a365dc4f17/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/bfec23ddaa228e438cf4234beba5ba12a61b78019c97475b5335848319a98b07/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/57351dc868f01750b3382f3cb46833ef5d00fadaaa738440b0cc9d2b7ea09766/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5845671cbcf687437a42b51b41268919db3120be7c3628aff5e6c4aa44fdad65/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b72c83aa9270995524f348df660c46b0616db89c270e4b46f941c446a5d9dc78/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0e48556d325c385ed4c6623fe943e3eeb54cf65c8baac0b08a74ede73b2097ce/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5350e1055f99f91f2cd010ae51c97ec0063f0082f2658daf270c404f88fbcead/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/64bbd45c79edf93931015014f2361013538f989dbba7f0a28625eb1af54da470/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/1cb6781600b8612a91e2d40adb6a910952287a0329df5da90e4898a0e4fc8a79/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.513624 5859 manager.go:217] Machine: {Timestamp:2026-11-02 21:09:20.512839207 +0000 UTC m=+0.061088980 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199797 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec218698333d06c45c3c8121c41a55a0 SystemUUID:ec218698-333d-06c4-5c3c-8121c41a55a0 BootID:875c9d7b-914f-4da0-aa3f-a415de742a21 Filesystems:[{Device:/run/containers/storage/overlay-containers/f38ab114864eaabcc52b60dbc711268b3c03d52565f49d7cebc210cc018cf3b0/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61d0c582-d441-4746-8619-d43d8f4099fe/volumes/kubernetes.io~projected/kube-api-access-wl2sd DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6c292ae7ed7e5fef3a8598f3d77e87be1a96da1ac9cceedb143627c0215ae3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbdffb38-5e8e-4484-a77a-abaea4c5aca4/volumes/kubernetes.io~projected/kube-api-access-8vzks DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/635dec1c9094554e771216e5959ca4be43f1a9981325f54024414d7a9e8b4634/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:439 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:373 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/026c19eb2949abd9508c6e1b99daf5680eb5a05a8418744ab10cba30bc3fd264/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fddf083e-247f-4cdc-9722-3e6c2b43dd50/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:443 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~projected/kube-api-access-tk7q6 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4238a8dd-cf04-4f35-bb82-fce7996270a4/volumes/kubernetes.io~projected/kube-api-access-x8jds DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~projected/kube-api-access-hxjgd DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~projected/kube-api-access-kfs9k DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baf62419-1dfd-4003-937c-6f3a835bd88e/volumes/kubernetes.io~projected/kube-api-access-vqqq6 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eff3d924ba85cff135d93224bcd3c5faade1cd181a078c47f42751344c48b572/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67588178bd23332be0b035e529afe1bf2dba15e470f7915f0c92e26961dbf16b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~projected/kube-api-access-96bhz DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~projected/kube-api-access-xssrx DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbda9bc1-119f-4474-9cca-9204e084f4b6/volumes/kubernetes.io~projected/kube-api-access-8djpn DeviceMajor:0 DeviceMinor:446 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/018ea93d-84d6-4e86-98ce-e376674b7a12/volumes/kubernetes.io~projected/kube-api-access-87gcx DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e/volumes/kubernetes.io~projected/kube-api-access-2nkgf DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~projected/kube-api-access-sscd5 DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ce4c8d11ab44e2af80ec4936a5b2a6dab2bf8b162a9d91d9d7ab5f0a9a5bff3/userdata/shm DeviceMajor:0 DeviceMinor:447 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4e32c9811b9f72cb8719cf50b79330f218ae87db65082bd19e01da34bfc0263/userdata/shm DeviceMajor:0 DeviceMinor:449 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a9f10b241bc18c173beba16827fec3a2b4d78f7057df0306d0028d52677d61c/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b4e259cc18ab1136f5a173fd31e6f2714c07c768ae77f7f5b0b47beed12af67/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0899cbf8d444dccbba4368b83f557ccda9c65bc7f5919024e788242f7e54dd8b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3196b4607f8ba4fc65290e6cd04b7702b4baf285d354f3dd9f0c59c7ff2b680/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/992df832-d24f-4517-ab42-d47ff9f51b7e/volumes/kubernetes.io~projected/kube-api-access-ngfkf DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~projected/kube-api-access-sk29p DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93eb21de-12ad-49fb-b162-b53f3763d2ce/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/021723a2c141bc887b2df846b137b721f9aaa77d64fde3806c0c33c1fcb3baec/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97/volumes/kubernetes.io~projected/kube-api-access-5zpqr DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4d69e9e1bc2b06e58c5900650a73506dc26d3e98febba0dc0ccef6c295cfce1/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c812be04-0eac-4a4c-9456-60332f6fa93d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ea766a0888db7769dfee803fa37147348bd71cd5cb2c13ac4cc983c7bcf51b5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13c167afe078d74351ada894d732775a455c3f61bb22c10dd4eca3df46f41d5c/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04d04d90-6021-4329-b47d-a41847969eef/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0f84692-9d39-43f4-ac6a-c78d837e0107/volumes/kubernetes.io~projected/kube-api-access-75f8w DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/917f2edea2ed0eb01697b0d318db977fcf71448f0d6e39b088d4afd391f253a1/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0fd6aa79507b9dd9af85f126224545323b81b9390e91fb8748bc9a8e60245e8/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e77bc5a-0603-4b53-900d-c3335549d101/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes/kubernetes.io~projected/kube-api-access-j8ws4 DeviceMajor:0 DeviceMinor:379 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6/volumes/kubernetes.io~projected/kube-api-access-n7fdm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e84ee36e6aa020d99ab379eb9b2f1cb7bf33d1a41ba1af7cf1cc2195c9904bd/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0423cd6d-c791-4383-92cd-8b699aca0c0e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/efcdbaab-36bb-4f9a-9008-4e21d230deea/volumes/kubernetes.io~projected/kube-api-access-thd2l DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e07eb1f-6b9f-4626-ab55-efb294a3d136/volumes/kubernetes.io~projected/kube-api-access-phsrf DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2895103fdc3ff098ceb287ac4a1feba6d18e9b8ecab91a54eb4e4c3cf0e455be/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:026c19eb2949abd MacAddress:b2:9b:5c:e2:ac:a7 Speed:10000 Mtu:8901} {Name:0899cbf8d444dcc MacAddress:ee:7c:7b:fc:89:a8 Speed:10000 Mtu:8901} {Name:13c167afe078d74 MacAddress:82:f4:82:a3:cd:1e Speed:10000 Mtu:8901} {Name:3d8a8b9a5749323 MacAddress:7e:8e:1b:12:98:5b Speed:10000 Mtu:8901} {Name:4ce4c8d11ab44e2 MacAddress:ca:68:24:a3:65:e9 Speed:10000 Mtu:8901} {Name:67588178bd23332 MacAddress:7e:6d:eb:32:33:b2 Speed:10000 Mtu:8901} {Name:6e84ee36e6aa020 MacAddress:52:50:d1:3a:71:a5 Speed:10000 Mtu:8901} {Name:7a9f10b241bc18c MacAddress:5e:29:18:07:0c:a6 Speed:10000 Mtu:8901} {Name:a4e32c9811b9f72 MacAddress:ba:9e:32:db:e6:eb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:b8:e1:f7 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:e2:3c:c7:bc:68 Speed:0 Mtu:8901} {Name:c97dc72bc48f9fc MacAddress:fa:8f:e7:c0:dd:5d Speed:10000 Mtu:8901} {Name:e3196b4607f8ba4 MacAddress:1a:1e:c4:88:f0:ce Speed:10000 Mtu:8901} {Name:eff3d924ba85cff MacAddress:e2:78:ba:8a:8a:eb Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:b8:e1:f7 Speed:0 Mtu:9001} {Name:f38ab114864eaab MacAddress:72:36:18:3b:32:e2 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:16:bf:ad:da:80:57 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:f9:27:bb:59:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514416 5859 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514478 5859 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514771 5859 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514793 5859 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-1.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514906 5859 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514914 5859 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514926 5859 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.514939 5859 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515012 5859 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515188 5859 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515288 5859 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515323 5859 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515332 5859 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515347 5859 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515356 5859 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:09:20.516832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.515363 5859 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:09:20.518636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.518617 5859 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:09:20.518636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.518635 5859 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:09:20.519123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.519104 5859 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:09:20.522095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522081 5859 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:09:20.522095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522094 5859 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:09:20.522208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522188 5859 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:09:20.522371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522360 5859 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:09:20.522401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522388 5859 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:09:20.522476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522468 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522481 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522487 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522493 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522499 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522504 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522509 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:09:20.522514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522515 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522521 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522528 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522544 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522553 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522577 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:09:20.522720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.522583 5859 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:09:20.525709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.525693 5859 server.go:1263] "Started kubelet" Nov 02 21:09:20.525847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.525773 5859 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:09:20.526366 ip-10-0-137-1 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:09:20.527206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.527188 5859 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:09:20.531161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.525900 5859 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:09:20.531244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.531230 5859 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:09:20.531855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.531512 5859 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:09:20.532538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.532518 5859 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:09:20.533477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.533306 5859 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:09:20.533477 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.533469 5859 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:09:20.533684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.533506 5859 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:09:20.533684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.533515 5859 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:09:20.533869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.533847 5859 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:09:20.534018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.533972 5859 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:03:10 +0000 UTC" deadline="2028-07-18 14:02:23.80694556 +0000 UTC" Nov 02 21:09:20.534018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.534009 5859 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14968h53m3.272939246s" Nov 02 21:09:20.534915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.534851 5859 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:09:20.534915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.534870 5859 factory.go:55] Registering systemd factory Nov 02 21:09:20.534915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.534879 5859 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:09:20.535682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.535162 5859 factory.go:153] Registering CRI-O factory Nov 02 21:09:20.535682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.535177 5859 factory.go:223] Registration of the crio container factory successfully Nov 02 21:09:20.535682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.535195 5859 factory.go:103] Registering Raw factory Nov 02 21:09:20.535682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.535212 5859 manager.go:1199] Started watching for new ooms in manager Nov 02 21:09:20.537703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.535553 5859 manager.go:319] Starting recovery of all containers Nov 02 21:09:20.538816 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538777 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97" volumeName="kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538821 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e07eb1f-6b9f-4626-ab55-efb294a3d136" volumeName="kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538838 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538852 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538865 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0423cd6d-c791-4383-92cd-8b699aca0c0e" volumeName="kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538878 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0423cd6d-c791-4383-92cd-8b699aca0c0e" volumeName="kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd" seLinuxMountContext="" Nov 02 21:09:20.538907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538892 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538946 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538977 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97" volumeName="kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.538994 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539021 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539034 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539046 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539060 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97" volumeName="kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539075 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539087 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539098 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539110 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539123 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539135 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539149 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf62419-1dfd-4003-937c-6f3a835bd88e" volumeName="kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539163 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04d04d90-6021-4329-b47d-a41847969eef" volumeName="kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539175 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539197 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8bc690-2b44-4fd3-a2d2-62da0962ec4e" volumeName="kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539209 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls" seLinuxMountContext="" Nov 02 21:09:20.539220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539222 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539252 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539267 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" volumeName="kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539284 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539296 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04d04d90-6021-4329-b47d-a41847969eef" volumeName="kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539307 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539334 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539346 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf62419-1dfd-4003-937c-6f3a835bd88e" volumeName="kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539357 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f84692-9d39-43f4-ac6a-c78d837e0107" volumeName="kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539368 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2" volumeName="kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539385 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbdffb38-5e8e-4484-a77a-abaea4c5aca4" volumeName="kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539399 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61d0c582-d441-4746-8619-d43d8f4099fe" volumeName="kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539413 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="018ea93d-84d6-4e86-98ce-e376674b7a12" volumeName="kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539426 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539440 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4238a8dd-cf04-4f35-bb82-fce7996270a4" volumeName="kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539451 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539464 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539475 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539487 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbda9bc1-119f-4474-9cca-9204e084f4b6" volumeName="kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539500 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539511 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8bc690-2b44-4fd3-a2d2-62da0962ec4e" volumeName="kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539524 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539536 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539547 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2" volumeName="kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539558 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2" volumeName="kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539570 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efcdbaab-36bb-4f9a-9008-4e21d230deea" volumeName="kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539583 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539593 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8bc690-2b44-4fd3-a2d2-62da0962ec4e" volumeName="kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539602 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539613 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539627 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fddf083e-247f-4cdc-9722-3e6c2b43dd50" volumeName="kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539640 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539647 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539654 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e07eb1f-6b9f-4626-ab55-efb294a3d136" volumeName="kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539661 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539671 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539683 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a8bc690-2b44-4fd3-a2d2-62da0962ec4e" volumeName="kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539691 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539698 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539709 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539729 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539735 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992df832-d24f-4517-ab42-d47ff9f51b7e" volumeName="kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539746 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf62419-1dfd-4003-937c-6f3a835bd88e" volumeName="kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539757 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539764 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e07eb1f-6b9f-4626-ab55-efb294a3d136" volumeName="kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539771 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539779 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539785 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539793 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539800 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="018ea93d-84d6-4e86-98ce-e376674b7a12" volumeName="kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539807 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539814 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992df832-d24f-4517-ab42-d47ff9f51b7e" volumeName="kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539823 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992df832-d24f-4517-ab42-d47ff9f51b7e" volumeName="kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539834 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539846 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539853 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539860 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf62419-1dfd-4003-937c-6f3a835bd88e" volumeName="kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539867 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539876 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c812be04-0eac-4a4c-9456-60332f6fa93d" volumeName="kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539883 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0f84692-9d39-43f4-ac6a-c78d837e0107" volumeName="kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539891 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efcdbaab-36bb-4f9a-9008-4e21d230deea" volumeName="kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539900 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4238a8dd-cf04-4f35-bb82-fce7996270a4" volumeName="kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539910 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e07eb1f-6b9f-4626-ab55-efb294a3d136" volumeName="kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539917 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992df832-d24f-4517-ab42-d47ff9f51b7e" volumeName="kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539945 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539970 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfa5c3f8-0ffe-4e56-adb0-ad12287596c6" volumeName="kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539983 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbdffb38-5e8e-4484-a77a-abaea4c5aca4" volumeName="kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.539990 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbdffb38-5e8e-4484-a77a-abaea4c5aca4" volumeName="kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540019 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540028 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540036 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540046 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e07eb1f-6b9f-4626-ab55-efb294a3d136" volumeName="kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540063 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" volumeName="kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540071 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540080 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eb21de-12ad-49fb-b162-b53f3763d2ce" volumeName="kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540087 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" volumeName="kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540094 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540101 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1" volumeName="kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540109 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e77bc5a-0603-4b53-900d-c3335549d101" volumeName="kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540117 5859 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97" volumeName="kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr" seLinuxMountContext="" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540129 5859 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:09:20.540146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.540135 5859 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:09:20.561287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.561245 5859 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:09:20.562620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.562602 5859 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:09:20.562707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.562623 5859 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:09:20.562707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.562642 5859 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:09:20.562707 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.562685 5859 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:09:20.572259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.572230 5859 generic.go:356] "Generic (PLEG): container finished" podID="93eb21de-12ad-49fb-b162-b53f3763d2ce" containerID="f7295ed6a54d35a10bc5d08203238436d96f3ac4f6f7a85ccb6379adb94eedcd" exitCode=0 Nov 02 21:09:20.579795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579769 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="b32e183fa1b65821e3f9f1a8e2970260188f5a68a9d159dad69e150a4d892d43" exitCode=0 Nov 02 21:09:20.579795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579796 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="79517b0601ba490bcdc91a160caa674ab723bdc131d1166335acfa6be05918fb" exitCode=0 Nov 02 21:09:20.579923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579803 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="e5efced5a740c8ad1d147972842c300bdd4c5de22deefa8fc3bbaa6eaa80318f" exitCode=0 Nov 02 21:09:20.579923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579809 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="535a4c60b5a8c6028889873a3591bc4fbc1e1ac58badb15fbd54c43636e50f41" exitCode=0 Nov 02 21:09:20.579923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579813 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="24abd1b375a8af4eada026668df62933526a613ad135a1266911278a31a59328" exitCode=0 Nov 02 21:09:20.579923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.579818 5859 generic.go:356] "Generic (PLEG): container finished" podID="baf62419-1dfd-4003-937c-6f3a835bd88e" containerID="1c9423bd8261afdb571a485668a302083e835b625b9b98ed664370b46a15cf03" exitCode=0 Nov 02 21:09:20.596516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.596497 5859 generic.go:356] "Generic (PLEG): container finished" podID="61e83f034a10ddc04c4aab6039382daa" containerID="9b2a1cc1da1138b2856dd99f3f8420ecc822e08765fe6ba3debb177ceac01f1a" exitCode=0 Nov 02 21:09:20.614335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.614309 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="431ffc7f069c226092e6138a9629410989a912c6271afedbfa5c9ac2acc46c76" exitCode=0 Nov 02 21:09:20.625922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.625891 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" exitCode=0 Nov 02 21:09:20.633801 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.633784 5859 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:09:20.648150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.648101 5859 manager.go:324] Recovery completed Nov 02 21:09:20.663620 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.663592 5859 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 21:09:20.669384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.669369 5859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:09:20.671859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.671844 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:09:20.671930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.671870 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:09:20.671930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.671880 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:09:20.674149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674131 5859 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:09:20.674149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674143 5859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:09:20.674149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674148 5859 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674168 5859 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674330 5859 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674343 5859 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674358 5859 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674363 5859 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674369 5859 policy_none.go:50] "Start" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674378 5859 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:09:20.674439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.674388 5859 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:09:20.675245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.675230 5859 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:09:20.675285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.675246 5859 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:09:20.675285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.675253 5859 policy_none.go:44] "Start" Nov 02 21:09:20.676453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.676438 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:09:20.676544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.676469 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:09:20.676544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.676483 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:09:20.679922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.679894 5859 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:09:20.683184 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683172 5859 manager.go:341] "Starting Device Plugin manager" Nov 02 21:09:20.683263 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.683201 5859 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:09:20.683263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683210 5859 server.go:87] "Starting device plugin registration server" Nov 02 21:09:20.683460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683446 5859 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:09:20.683520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683460 5859 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:09:20.683562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683533 5859 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:09:20.683657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683624 5859 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:09:20.683738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683669 5859 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:09:20.683738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683676 5859 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:09:20.683821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.683729 5859 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:09:20.683655338Z","UUID":"21209d97-572f-458f-98fa-68b83b432af8","Handler":null,"Name":"","Endpoint":""} Nov 02 21:09:20.686081 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.686005 5859 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:09:20.686200 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.686118 5859 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-1.us-west-2.compute.internal\" not found" Nov 02 21:09:20.686661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.686459 5859 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:09:20.686661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.686497 5859 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:09:20.713756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.713734 5859 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:09:20.713828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.713782 5859 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:09:20.714435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.714421 5859 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:09:20.759572 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.759549 5859 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:09:20.786864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.786847 5859 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:09:20.787900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.787886 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:09:20.787989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.787911 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:09:20.787989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.787920 5859 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-1.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:09:20.787989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.787980 5859 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.806464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.806448 5859 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:09:20.863810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.863776 5859 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal"] Nov 02 21:09:20.864398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864370 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerStarted","Data":"c6e9b9d823f782260537ff1a3d8e447a82739560e6340666cadd161f289cc0c5"} Nov 02 21:09:20.864459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864404 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerDied","Data":"9b2a1cc1da1138b2856dd99f3f8420ecc822e08765fe6ba3debb177ceac01f1a"} Nov 02 21:09:20.864459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864419 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"61e83f034a10ddc04c4aab6039382daa","Type":"ContainerStarted","Data":"3b751f014f9c1efd9a8c2853bbe4ffd70387b7bb73f536f27c7009a3d5032472"} Nov 02 21:09:20.864459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864430 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"aedbdf8e991546e8774cd514478921e1","Type":"ContainerStarted","Data":"a1bf97d65886c9b9f988364a438aadeb5135c297690ef2c8b3ee3193d17ce8af"} Nov 02 21:09:20.864459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864440 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" event={"ID":"aedbdf8e991546e8774cd514478921e1","Type":"ContainerStarted","Data":"fa96d49299b6bc0feea52a6359d72007d6ce0e0da6598e7d5ee2e25a94c7e6d4"} Nov 02 21:09:20.864600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864470 5859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.864600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.864553 5859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.912339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.912291 5859 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.912339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.912338 5859 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.922672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.922654 5859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:09:20.922746 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.922690 5859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:20.923189 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:20.923175 5859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:09:20.923256 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:20.923212 5859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.015878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.015858 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.015967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.015883 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.015967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.015900 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116393 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116421 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116437 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116470 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aedbdf8e991546e8774cd514478921e1-config\") pod \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"aedbdf8e991546e8774cd514478921e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116502 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.116620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.116503 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e83f034a10ddc04c4aab6039382daa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" (UID: \"61e83f034a10ddc04c4aab6039382daa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.516008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.515988 5859 apiserver.go:52] "Watching apiserver" Nov 02 21:09:21.613445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.613424 5859 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:09:21.614051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.614032 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-99xw5","openshift-insights/insights-runtime-extractor-tgsfm","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-6phbr","openshift-ingress-canary/ingress-canary-hdm6g","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2","openshift-monitoring/thanos-querier-d55f57bf6-46r5k","openshift-multus/multus-ssnrt","openshift-monitoring/metrics-server-7b797d4856-8xgjr","openshift-ovn-kubernetes/ovnkube-node-rsvj8","openshift-cluster-node-tuning-operator/tuned-8pk9f","kube-system/global-pull-secret-syncer-xm28p","openshift-network-diagnostics/network-check-target-pt5zs","kube-system/konnectivity-agent-vbbpt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp","openshift-image-registry/node-ca-8spgj","openshift-multus/network-metrics-daemon-xk8mq","openshift-network-operator/iptables-alerter-fzp9j","kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal","openshift-monitoring/prometheus-k8s-0","openshift-dns/dns-default-q7knd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69","openshift-multus/multus-additional-cni-plugins-kmm87","openshift-dns/node-resolver-9jbsf"] Nov 02 21:09:21.629422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.629368 5859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.629518 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.629504 5859 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.636759 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.636743 5859 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:09:21.663580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.663565 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\"" Nov 02 21:09:21.664338 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664320 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:09:21.664411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664323 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.664411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664341 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:09:21.664411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664353 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.664411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664400 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.664524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664451 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.664755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.664744 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:09:21.665124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665112 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\"" Nov 02 21:09:21.665154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665136 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\"" Nov 02 21:09:21.665546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665534 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\"" Nov 02 21:09:21.665687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665664 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:09:21.665781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665729 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:09:21.665850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665834 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\"" Nov 02 21:09:21.665919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665881 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\"" Nov 02 21:09:21.665992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.665927 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\"" Nov 02 21:09:21.666353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666333 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:09:21.666353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666342 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:09:21.666439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666367 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.666439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666379 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.666439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666388 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.666439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666381 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.666439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666436 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:09:21.666644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.666628 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:09:21.668504 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.668490 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.669809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669758 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:09:21.669809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669803 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\"" Nov 02 21:09:21.669938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669924 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.670017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669936 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.670017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669964 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.670017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669977 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.670017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.669996 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\"" Nov 02 21:09:21.670445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670427 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:09:21.670547 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670471 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.670618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670573 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:09:21.670697 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670673 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.670766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670754 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\"" Nov 02 21:09:21.670903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670887 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:09:21.670971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670906 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:09:21.670971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670910 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.670971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670914 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.670971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670914 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\"" Nov 02 21:09:21.671102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.670980 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:09:21.671191 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.671175 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\"" Nov 02 21:09:21.671369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.671358 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:09:21.716831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.716815 5859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:09:21.716911 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:21.716851 5859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.717163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.717152 5859 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:09:21.717198 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:21.717180 5859 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-1.us-west-2.compute.internal" Nov 02 21:09:21.720698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720668 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.720783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720701 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.720783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720721 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.720783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720742 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.720783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720764 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.720971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720787 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.720971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720810 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.720971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720834 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.720971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720842 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.720971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720906 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.720980 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721018 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721029 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-audit-log\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721106 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721129 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.721148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721146 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721160 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721174 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721192 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f8w\" (UniqueName: \"kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721210 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721223 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdbaab-36bb-4f9a-9008-4e21d230deea-tmp-dir\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721275 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-data-volume\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721302 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.721388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721326 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721391 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721396 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721424 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-tuned\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721439 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721460 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721475 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721509 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721550 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721581 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721604 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721621 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.721692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721671 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721751 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721774 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thd2l\" (UniqueName: \"kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721795 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721821 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721820 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-daemon-config\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721928 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04d04d90-6021-4329-b47d-a41847969eef-konnectivity-ca\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.721975 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722009 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722048 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722033 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722146 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722170 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722194 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722233 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722235 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fddf083e-247f-4cdc-9722-3e6c2b43dd50-original-pull-secret\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.722272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722255 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722311 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722338 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722352 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722378 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722526 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722551 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722580 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722616 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722637 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722658 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jds\" (UniqueName: \"kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722689 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722709 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722734 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722754 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722775 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.722824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722796 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722883 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.722973 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723004 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723053 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723076 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723101 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723126 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723198 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723283 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqq6\" (UniqueName: \"kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723309 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723333 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723357 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723383 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.723407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkgf\" (UniqueName: \"kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723474 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723500 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723522 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723543 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723572 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723594 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723619 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723627 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4238a8dd-cf04-4f35-bb82-fce7996270a4-serviceca\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723646 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.723811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723663 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04d04d90-6021-4329-b47d-a41847969eef-agent-certs\") pod \"konnectivity-agent-vbbpt\" (UID: \"04d04d90-6021-4329-b47d-a41847969eef\") " pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:09:21.724129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.723994 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.724129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724015 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.724129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724035 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-metrics-certs\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:21.724227 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724115 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.724227 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724163 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.724227 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724207 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.724357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724258 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzks\" (UniqueName: \"kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.724357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724300 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fdm\" (UniqueName: \"kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:21.724415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724393 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.724516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724494 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.724557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724534 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk29p\" (UniqueName: \"kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.724557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724554 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.724629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724575 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.724685 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724670 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gcx\" (UniqueName: \"kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.724734 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724693 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.724734 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724711 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.724734 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724731 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.724854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724781 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.724854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724789 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.724854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724798 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.724854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724814 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.724854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724834 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724883 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724914 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.724991 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725026 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725073 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725097 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725121 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-textfile\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725138 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjgd\" (UniqueName: \"kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:21.725228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725224 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725247 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-script-lib\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725253 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725289 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725320 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725344 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725372 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725419 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725419 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cni-binary-copy\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725460 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725486 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725521 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725595 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725640 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725667 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725667 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baf62419-1dfd-4003-937c-6f3a835bd88e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725688 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725720 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725780 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.725870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725838 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725909 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.725995 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726037 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726059 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726085 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726100 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/018ea93d-84d6-4e86-98ce-e376674b7a12-iptables-alerter-script\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726153 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpqr\" (UniqueName: \"kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.726215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726171 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-env-overrides\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726243 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xssrx\" (UniqueName: \"kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726287 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726310 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726329 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726348 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726368 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726417 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.726506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726482 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726600 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726621 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726655 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovn-node-metrics-cert\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726664 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726704 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726733 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726761 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726797 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726823 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726851 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.726874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726853 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e07eb1f-6b9f-4626-ab55-efb294a3d136-ovnkube-config\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726885 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsrf\" (UniqueName: \"kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726921 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.726976 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727018 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c812be04-0eac-4a4c-9456-60332f6fa93d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727025 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727073 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727092 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727114 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfkf\" (UniqueName: \"kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727239 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.727266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727271 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727311 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk7q6\" (UniqueName: \"kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727332 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727352 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727373 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727397 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2sd\" (UniqueName: \"kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727424 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727451 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96bhz\" (UniqueName: \"kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727475 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727475 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-tmp\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727500 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727546 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e77bc5a-0603-4b53-900d-c3335549d101-tmp\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.727686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.727557 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-tmp-dir\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.733902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.733881 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:09:21.733902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.733883 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 21:09:21.734027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.733929 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:09:21.734027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.733990 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:09:21.734267 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734250 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\"" Nov 02 21:09:21.734333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734270 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:09:21.734333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734286 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:09:21.734424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734340 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nwd24\"" Nov 02 21:09:21.734424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734351 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bhgh7\"" Nov 02 21:09:21.734424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734397 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dzqlm\"" Nov 02 21:09:21.734424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734397 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ptqqmo5n63c4\"" Nov 02 21:09:21.734583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734449 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:09:21.734583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734454 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:09:21.734583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734452 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:09:21.734583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734480 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:09:21.734583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734509 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:09:21.734738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734618 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 21:09:21.734738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734654 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:09:21.734738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734664 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:09:21.734738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734692 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:09:21.734738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734728 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:09:21.734886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734777 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:09:21.734886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734883 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:09:21.734984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734940 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:09:21.735017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734996 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:09:21.735017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.734999 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:09:21.735105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735000 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:09:21.735105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735018 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7dlcpn653nlg\"" Nov 02 21:09:21.735105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735062 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 21:09:21.735105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735104 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735141 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735149 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735165 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735180 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sb2rf\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735185 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2gs96mbktstv5\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735186 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735188 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735253 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\"" Nov 02 21:09:21.735263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735261 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735266 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbphx\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735263 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735285 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jxzwg\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735297 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b7xvl\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735338 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735385 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735404 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735453 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 21:09:21.735519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735495 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:09:21.735795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735535 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lnmnx\"" Nov 02 21:09:21.735795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735567 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 21:09:21.735795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735715 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.735926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735841 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0f84692-9d39-43f4-ac6a-c78d837e0107-cert\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:09:21.736001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.735937 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.736078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.736051 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.736117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.736076 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.736443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.736430 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.737628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.737603 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.737693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.737624 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.737693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.737686 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992df832-d24f-4517-ab42-d47ff9f51b7e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.737750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.737700 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-tls\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.737750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.737716 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.741902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.741881 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.742023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.741889 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.742023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.741920 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.742023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.741928 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-tls\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.742187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.742145 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.742297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.742283 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0423cd6d-c791-4383-92cd-8b699aca0c0e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:21.742407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.742392 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.743096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743078 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.743162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743122 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.743162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743121 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.743162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743138 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.743162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743143 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.743162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743155 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.743322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.743219 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-grpc-tls\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.744416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744380 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.744486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744417 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1e77bc5a-0603-4b53-900d-c3335549d101-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.744486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744416 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.744486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744421 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.744486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744448 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.744667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.744410 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-metrics-tls\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.745711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.745692 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-secret-metrics-server-client-certs\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.745781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.745736 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-client-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.764281 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764261 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:09:21.764281 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764262 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:09:21.764748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764728 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.764748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764728 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 21:09:21.764944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764750 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.764944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764763 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:09:21.764944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764750 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.764944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.764920 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 21:09:21.765159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765014 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:09:21.765159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765080 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:09:21.765159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765141 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.765271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765219 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:09:21.765271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765219 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:09:21.765271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765231 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:09:21.765456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765440 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:09:21.765510 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765471 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 21:09:21.765568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.765549 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-metrics-client-ca\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.766790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.766750 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e77bc5a-0603-4b53-900d-c3335549d101-metrics-client-ca\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.766845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.766792 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992df832-d24f-4517-ab42-d47ff9f51b7e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.766877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.766843 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.766877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.766866 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.768120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.768103 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.768156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.768103 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.772146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.772123 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.772230 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.772198 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-metrics-server-audit-profiles\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.772290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.772252 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.772325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.772293 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.773465 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.773442 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.773531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.773491 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.774652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.774634 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-config-volume\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.774652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.774646 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782126 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782551 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jds\" (UniqueName: \"kubernetes.io/projected/4238a8dd-cf04-4f35-bb82-fce7996270a4-kube-api-access-x8jds\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782719 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f8w\" (UniqueName: \"kubernetes.io/projected/d0f84692-9d39-43f4-ac6a-c78d837e0107-kube-api-access-75f8w\") pod \"ingress-canary-hdm6g\" (UID: \"d0f84692-9d39-43f4-ac6a-c78d837e0107\") " pod="openshift-ingress-canary/ingress-canary-hdm6g" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782731 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782720 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782747 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.782971 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk29p\" (UniqueName: \"kubernetes.io/projected/c812be04-0eac-4a4c-9456-60332f6fa93d-kube-api-access-sk29p\") pod \"kube-state-metrics-5dc8d7d89b-cs4v2\" (UID: \"c812be04-0eac-4a4c-9456-60332f6fa93d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cs4v2" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783013 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djpn\" (UniqueName: \"kubernetes.io/projected/cbda9bc1-119f-4474-9cca-9204e084f4b6-kube-api-access-8djpn\") pod \"network-check-target-pt5zs\" (UID: \"cbda9bc1-119f-4474-9cca-9204e084f4b6\") " pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783140 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gcx\" (UniqueName: \"kubernetes.io/projected/018ea93d-84d6-4e86-98ce-e376674b7a12-kube-api-access-87gcx\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783141 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkgf\" (UniqueName: \"kubernetes.io/projected/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-kube-api-access-2nkgf\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783275 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqq6\" (UniqueName: \"kubernetes.io/projected/baf62419-1dfd-4003-937c-6f3a835bd88e-kube-api-access-vqqq6\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783288 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzks\" (UniqueName: \"kubernetes.io/projected/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-kube-api-access-8vzks\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783322 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fdm\" (UniqueName: \"kubernetes.io/projected/bfa5c3f8-0ffe-4e56-adb0-ad12287596c6-kube-api-access-n7fdm\") pod \"network-metrics-daemon-xk8mq\" (UID: \"bfa5c3f8-0ffe-4e56-adb0-ad12287596c6\") " pod="openshift-multus/network-metrics-daemon-xk8mq" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783516 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xssrx\" (UniqueName: \"kubernetes.io/projected/1e77bc5a-0603-4b53-900d-c3335549d101-kube-api-access-xssrx\") pod \"thanos-querier-d55f57bf6-46r5k\" (UID: \"1e77bc5a-0603-4b53-900d-c3335549d101\") " pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783578 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk7q6\" (UniqueName: \"kubernetes.io/projected/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-kube-api-access-tk7q6\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783648 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsrf\" (UniqueName: \"kubernetes.io/projected/7e07eb1f-6b9f-4626-ab55-efb294a3d136-kube-api-access-phsrf\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783648 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2sd\" (UniqueName: \"kubernetes.io/projected/61d0c582-d441-4746-8619-d43d8f4099fe-kube-api-access-wl2sd\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783730 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpqr\" (UniqueName: \"kubernetes.io/projected/7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97-kube-api-access-5zpqr\") pod \"dns-default-q7knd\" (UID: \"7cd6e1a3-ff2f-4b77-97a7-33a39ab3eb97\") " pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783743 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thd2l\" (UniqueName: \"kubernetes.io/projected/efcdbaab-36bb-4f9a-9008-4e21d230deea-kube-api-access-thd2l\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783819 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfkf\" (UniqueName: \"kubernetes.io/projected/992df832-d24f-4517-ab42-d47ff9f51b7e-kube-api-access-ngfkf\") pod \"openshift-state-metrics-7bd7489957-v2l69\" (UID: \"992df832-d24f-4517-ab42-d47ff9f51b7e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v2l69" Nov 02 21:09:21.783980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.783982 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96bhz\" (UniqueName: \"kubernetes.io/projected/93eb21de-12ad-49fb-b162-b53f3763d2ce-kube-api-access-96bhz\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.784649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.784069 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.784649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.784131 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjgd\" (UniqueName: \"kubernetes.io/projected/0423cd6d-c791-4383-92cd-8b699aca0c0e-kube-api-access-hxjgd\") pod \"monitoring-plugin-77f4f76594-99xw5\" (UID: \"0423cd6d-c791-4383-92cd-8b699aca0c0e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:21.784649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.784234 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs9k\" (UniqueName: \"kubernetes.io/projected/1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1-kube-api-access-kfs9k\") pod \"metrics-server-7b797d4856-8xgjr\" (UID: \"1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1\") " pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:21.786032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.786017 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:21.787242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.787227 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.788786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.788765 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828138 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828195 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828227 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828286 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-device-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828334 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828361 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-k8s-cni-cncf-io\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.828445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828432 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4238a8dd-cf04-4f35-bb82-fce7996270a4-host\") pod \"node-ca-8spgj\" (UID: \"4238a8dd-cf04-4f35-bb82-fce7996270a4\") " pod="openshift-image-registry/node-ca-8spgj" Nov 02 21:09:21.828864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828557 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-multus-certs\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.828864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828808 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.828864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828849 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828879 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828909 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828938 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.828981 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.829027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829019 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829049 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829080 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829111 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829149 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829179 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829210 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829244 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829274 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829305 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829337 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829363 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829406 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829436 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829467 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829502 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829528 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829561 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.829611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829593 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829626 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829658 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829662 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-kubelet\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829685 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829734 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-system-cni-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829744 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-slash\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829768 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829795 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829809 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-node-log\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829833 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829862 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829878 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-socket-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829897 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829926 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829986 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829995 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-conf-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830020 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830049 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-cnibin\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830050 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830102 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.830112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830110 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-host\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830125 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830156 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830164 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/018ea93d-84d6-4e86-98ce-e376674b7a12-host-slash\") pod \"iptables-alerter-fzp9j\" (UID: \"018ea93d-84d6-4e86-98ce-e376674b7a12\") " pod="openshift-network-operator/iptables-alerter-fzp9j" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830187 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830210 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830216 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830263 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830268 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-var-lib-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830320 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-hostroot\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830341 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-cnibin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830377 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-registration-dir\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830404 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-socket-dir-parent\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830422 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-netd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830452 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-ovn\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830470 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-netns\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830507 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-log-socket\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830510 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-kubernetes\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830572 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-dbus\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830577 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysconfig\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830625 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-root\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830639 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-kubelet\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830685 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-cni-bin\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830701 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-sys-fs\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830760 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-multus-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830793 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-modprobe-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830817 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baf62419-1dfd-4003-937c-6f3a835bd88e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmm87\" (UID: \"baf62419-1dfd-4003-937c-6f3a835bd88e\") " pod="openshift-multus/multus-additional-cni-plugins-kmm87" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830846 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830864 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8bc690-2b44-4fd3-a2d2-62da0962ec4e-crio-socket\") pod \"insights-runtime-extractor-tgsfm\" (UID: \"7a8bc690-2b44-4fd3-a2d2-62da0962ec4e\") " pod="openshift-insights/insights-runtime-extractor-tgsfm" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830878 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-systemd\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830906 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-etc-kubernetes\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830909 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830945 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830980 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-multus\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.830992 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831030 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/efcdbaab-36bb-4f9a-9008-4e21d230deea-hosts-file\") pod \"node-resolver-9jbsf\" (UID: \"efcdbaab-36bb-4f9a-9008-4e21d230deea\") " pod="openshift-dns/node-resolver-9jbsf" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831076 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-systemd\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831195 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-conf\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-run-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.829928 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-var-lib-cni-bin\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831302 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-system-cni-dir\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831322 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831348 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-run\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831368 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831400 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831430 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831445 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-lib-modules\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831457 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831488 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-sys\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831538 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93eb21de-12ad-49fb-b162-b53f3763d2ce-node-exporter-wtmp\") pod \"node-exporter-6phbr\" (UID: \"93eb21de-12ad-49fb-b162-b53f3763d2ce\") " pod="openshift-monitoring/node-exporter-6phbr" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831569 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831591 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-var-lib-kubelet\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831657 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61d0c582-d441-4746-8619-d43d8f4099fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-nxpxp\" (UID: \"61d0c582-d441-4746-8619-d43d8f4099fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nxpxp" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831712 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-etc-openvswitch\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831729 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831750 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbdffb38-5e8e-4484-a77a-abaea4c5aca4-host-run-netns\") pod \"multus-ssnrt\" (UID: \"fbdffb38-5e8e-4484-a77a-abaea4c5aca4\") " pod="openshift-multus/multus-ssnrt" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831774 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-systemd-units\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831838 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-sys\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831865 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e07eb1f-6b9f-4626-ab55-efb294a3d136-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsvj8\" (UID: \"7e07eb1f-6b9f-4626-ab55-efb294a3d136\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831912 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fddf083e-247f-4cdc-9722-3e6c2b43dd50-kubelet-config\") pod \"global-pull-secret-syncer-xm28p\" (UID: \"fddf083e-247f-4cdc-9722-3e6c2b43dd50\") " pod="kube-system/global-pull-secret-syncer-xm28p" Nov 02 21:09:21.832614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:21.831943 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2-etc-sysctl-d\") pod \"tuned-8pk9f\" (UID: \"e6c7a3bc-93b3-4c1a-8c7f-8a36b7ef55f2\") " pod="openshift-cluster-node-tuning-operator/tuned-8pk9f" Nov 02 21:09:22.747367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:22.747332 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-99xw5" Nov 02 21:09:22.940813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:22.940782 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:23.774859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:23.774831 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:23.791813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:23.791795 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:23.864699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:23.864678 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7knd" Nov 02 21:09:24.859149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:24.859124 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:25.150743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:25.150673 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d55f57bf6-46r5k" Nov 02 21:09:25.942630 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:25.942598 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:09:30.179185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:30.179162 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:30.179719 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:30.179338 5859 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:09:33.624448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:33.624360 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsvj8" Nov 02 21:09:34.925185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:34.925158 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b797d4856-8xgjr" Nov 02 21:09:43.580237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580205 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:43.580695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580605 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="alertmanager" containerID="cri-o://39ab7919b721f6f667a78fea1f155f20c9af38b93279fa0c0e9e516a9b04be44" gracePeriod=120 Nov 02 21:09:43.580695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580643 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-web" containerID="cri-o://5d7676560b535c3574a37c847bad94e8b8571b5826fb8ae875279a721531a277" gracePeriod=120 Nov 02 21:09:43.580798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580747 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="prom-label-proxy" containerID="cri-o://1024676d51ec5ed833b1571d6dd5b2f4722fda9130472d355f19eefae878b686" gracePeriod=120 Nov 02 21:09:43.580798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580782 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy" containerID="cri-o://d58b4613479d46e8ade69d274ec5709efc830ebd5e355c48fa6b92fabe3d96e9" gracePeriod=120 Nov 02 21:09:43.580886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580794 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="config-reloader" containerID="cri-o://bdfb90f0780c2a7956f95f9decc4c8ab6ad529ddf53e57119c96f092f486dce6" gracePeriod=120 Nov 02 21:09:43.580886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.580810 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-metric" containerID="cri-o://6bd2d593491bd89d6bfa0629d5300f76f8caaab635781d5e66c9f42daa5c05cf" gracePeriod=120 Nov 02 21:09:43.709695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709670 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="1024676d51ec5ed833b1571d6dd5b2f4722fda9130472d355f19eefae878b686" exitCode=0 Nov 02 21:09:43.709695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709691 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="d58b4613479d46e8ade69d274ec5709efc830ebd5e355c48fa6b92fabe3d96e9" exitCode=0 Nov 02 21:09:43.709695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709698 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="bdfb90f0780c2a7956f95f9decc4c8ab6ad529ddf53e57119c96f092f486dce6" exitCode=0 Nov 02 21:09:43.709695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709707 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="39ab7919b721f6f667a78fea1f155f20c9af38b93279fa0c0e9e516a9b04be44" exitCode=0 Nov 02 21:09:43.709902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709741 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"1024676d51ec5ed833b1571d6dd5b2f4722fda9130472d355f19eefae878b686"} Nov 02 21:09:43.709902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709812 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"d58b4613479d46e8ade69d274ec5709efc830ebd5e355c48fa6b92fabe3d96e9"} Nov 02 21:09:43.709902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709838 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"bdfb90f0780c2a7956f95f9decc4c8ab6ad529ddf53e57119c96f092f486dce6"} Nov 02 21:09:43.709902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:43.709865 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"39ab7919b721f6f667a78fea1f155f20c9af38b93279fa0c0e9e516a9b04be44"} Nov 02 21:09:44.718112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.718089 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="6bd2d593491bd89d6bfa0629d5300f76f8caaab635781d5e66c9f42daa5c05cf" exitCode=0 Nov 02 21:09:44.718112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.718109 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerID="5d7676560b535c3574a37c847bad94e8b8571b5826fb8ae875279a721531a277" exitCode=0 Nov 02 21:09:44.718434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.718164 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"6bd2d593491bd89d6bfa0629d5300f76f8caaab635781d5e66c9f42daa5c05cf"} Nov 02 21:09:44.718434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.718253 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"5d7676560b535c3574a37c847bad94e8b8571b5826fb8ae875279a721531a277"} Nov 02 21:09:44.827873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.827853 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:44.893756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893706 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.893756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893736 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.893904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893764 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.893904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893793 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.893904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893830 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.893904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893883 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893934 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.893992 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894028 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894064 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894101 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894139 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.894386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894172 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") pod \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\" (UID: \"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7\") " Nov 02 21:09:44.895097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.894843 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:44.895608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.895580 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:44.896085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.896062 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:44.896678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.896653 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "kube-api-access-sscd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:44.897192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897055 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.897192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897153 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:44.897430 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897408 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.897866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897848 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.897993 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897970 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:44.898074 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.897995 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.898321 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.898296 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.901118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.901093 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.907581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.907557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config" pod "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" (UID: "b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:44.994910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994884 5859 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.994910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994906 5859 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-web-config\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994917 5859 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-cluster-tls-config\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994926 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-metrics-client-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994937 5859 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-main-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994946 5859 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-tls-assets\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994969 5859 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-main-db\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994978 5859 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994986 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscd5\" (UniqueName: \"kubernetes.io/projected/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-kube-api-access-sscd5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.994995 5859 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.995004 5859 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.995013 5859 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:44.995064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:44.995024 5859 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7-config-out\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:45.724740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.724703 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7","Type":"ContainerDied","Data":"3d8a8b9a5749323109d075bbf01e5a648e633314a3ec33a54788a89a6c73fd88"} Nov 02 21:09:45.724740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.724731 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:45.725137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.724793 5859 scope.go:122] "RemoveContainer" containerID="1024676d51ec5ed833b1571d6dd5b2f4722fda9130472d355f19eefae878b686" Nov 02 21:09:45.732406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.732388 5859 scope.go:122] "RemoveContainer" containerID="6bd2d593491bd89d6bfa0629d5300f76f8caaab635781d5e66c9f42daa5c05cf" Nov 02 21:09:45.738754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.738737 5859 scope.go:122] "RemoveContainer" containerID="d58b4613479d46e8ade69d274ec5709efc830ebd5e355c48fa6b92fabe3d96e9" Nov 02 21:09:45.744797 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.744780 5859 scope.go:122] "RemoveContainer" containerID="5d7676560b535c3574a37c847bad94e8b8571b5826fb8ae875279a721531a277" Nov 02 21:09:45.752700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.752682 5859 scope.go:122] "RemoveContainer" containerID="bdfb90f0780c2a7956f95f9decc4c8ab6ad529ddf53e57119c96f092f486dce6" Nov 02 21:09:45.759105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.759090 5859 scope.go:122] "RemoveContainer" containerID="39ab7919b721f6f667a78fea1f155f20c9af38b93279fa0c0e9e516a9b04be44" Nov 02 21:09:45.765262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.765241 5859 scope.go:122] "RemoveContainer" containerID="431ffc7f069c226092e6138a9629410989a912c6271afedbfa5c9ac2acc46c76" Nov 02 21:09:45.925796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.925751 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:45.972336 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972316 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:45.972658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972641 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:45.972929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972916 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="prom-label-proxy" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972929 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="prom-label-proxy" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972939 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="alertmanager" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972944 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="alertmanager" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972966 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="init-config-reloader" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972971 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="init-config-reloader" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972986 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-web" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.972991 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-web" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973001 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="config-reloader" Nov 02 21:09:45.973005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973005 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="config-reloader" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973012 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973017 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973027 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-metric" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973033 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-metric" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973093 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="config-reloader" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973104 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973111 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-metric" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973121 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="prom-label-proxy" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973129 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="alertmanager" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973136 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="init-config-reloader" Nov 02 21:09:45.973303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.973146 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" containerName="kube-rbac-proxy-web" Nov 02 21:09:45.978424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:45.978408 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.028199 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028177 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:09:46.028285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028222 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:09:46.028285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028233 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbphx\"" Nov 02 21:09:46.028285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028234 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:09:46.028285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028222 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:09:46.028285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028274 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:09:46.028464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028291 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:09:46.028501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028468 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:09:46.028651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.028637 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:09:46.074962 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.074930 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:09:46.105281 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105263 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105290 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105308 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105324 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105453 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105503 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105484 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105505 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105578 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105622 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105609 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjnx\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-kube-api-access-dtjnx\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105629 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105652 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105677 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.105761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.105749 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.132152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.132128 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:46.206889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.206815 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.206889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.206860 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjnx\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-kube-api-access-dtjnx\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.206891 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.206922 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.206969 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207003 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207039 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207066 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207091 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207118 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207152 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207187 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207221 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.207365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207243 5859 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:09:46.207610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207544 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.208126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.207906 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.208472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.208445 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.210192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210165 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.210356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210340 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.210498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210478 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.210498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210488 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.210839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210817 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.211013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.210994 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.211057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.211032 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.211588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.211568 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.212017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.212002 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.259496 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.259471 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjnx\" (UniqueName: \"kubernetes.io/projected/5fbe35f1-c94d-4036-b80f-449bf85d5b7a-kube-api-access-dtjnx\") pod \"alertmanager-main-0\" (UID: \"5fbe35f1-c94d-4036-b80f-449bf85d5b7a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.302886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.302866 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:09:46.568049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.568025 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7" path="/var/lib/kubelet/pods/b7c65bcf-b0a3-4109-8b9b-21e227f8b9e7/volumes" Nov 02 21:09:46.720311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:46.720282 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:09:46.761057 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:46.761028 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbe35f1_c94d_4036_b80f_449bf85d5b7a.slice/crio-7e8c746fd11ce04fc0ec6e1da55b81d2b8d39e06595e50df8dedf5da13e95374 WatchSource:0}: Error finding container 7e8c746fd11ce04fc0ec6e1da55b81d2b8d39e06595e50df8dedf5da13e95374: Status 404 returned error can't find the container with id 7e8c746fd11ce04fc0ec6e1da55b81d2b8d39e06595e50df8dedf5da13e95374 Nov 02 21:09:47.160769 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.160721 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbbpt" Nov 02 21:09:47.307309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.307285 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8589895978-xdksc"] Nov 02 21:09:47.310634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.310613 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.360060 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.360037 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 21:09:47.408149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408116 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 21:09:47.408295 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408197 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 21:09:47.408383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408217 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 21:09:47.408456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408242 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 21:09:47.408530 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408323 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 21:09:47.408623 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.408595 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c56kp\"" Nov 02 21:09:47.409086 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.409060 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8589895978-xdksc"] Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.414934 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.414995 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415035 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415061 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-federate-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415103 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415135 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-metrics-client-ca\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415155 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5qj\" (UniqueName: \"kubernetes.io/projected/53127e36-620e-41bb-940f-e4548cebc2ee-kube-api-access-wd5qj\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.415360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.415186 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-serving-certs-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.515985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.515928 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.515992 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516022 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516043 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-federate-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516073 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516102 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-metrics-client-ca\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516124 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5qj\" (UniqueName: \"kubernetes.io/projected/53127e36-620e-41bb-940f-e4548cebc2ee-kube-api-access-wd5qj\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516159 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-serving-certs-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.516936 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.516911 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-serving-certs-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.517114 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.517036 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-metrics-client-ca\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.517215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.517126 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.518912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.518847 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-telemeter-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.519056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.519033 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.519122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.519086 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-federate-client-tls\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.519254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.519236 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53127e36-620e-41bb-940f-e4548cebc2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.569335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.569307 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5qj\" (UniqueName: \"kubernetes.io/projected/53127e36-620e-41bb-940f-e4548cebc2ee-kube-api-access-wd5qj\") pod \"telemeter-client-8589895978-xdksc\" (UID: \"53127e36-620e-41bb-940f-e4548cebc2ee\") " pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.629358 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.629339 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" Nov 02 21:09:47.734272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.734244 5859 generic.go:356] "Generic (PLEG): container finished" podID="5fbe35f1-c94d-4036-b80f-449bf85d5b7a" containerID="ceabebc1ad01f6df274f2557ae75065731045247dca7e9e42181915f326261de" exitCode=0 Nov 02 21:09:47.734397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.734332 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerDied","Data":"ceabebc1ad01f6df274f2557ae75065731045247dca7e9e42181915f326261de"} Nov 02 21:09:47.734440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:47.734398 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"7e8c746fd11ce04fc0ec6e1da55b81d2b8d39e06595e50df8dedf5da13e95374"} Nov 02 21:09:48.061499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.049942 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8589895978-xdksc"] Nov 02 21:09:48.075451 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:48.075417 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53127e36_620e_41bb_940f_e4548cebc2ee.slice/crio-6ed304042c9800f8fe1e8654505c9838eed470312fde1bf72f423b1e9725ec42 WatchSource:0}: Error finding container 6ed304042c9800f8fe1e8654505c9838eed470312fde1bf72f423b1e9725ec42: Status 404 returned error can't find the container with id 6ed304042c9800f8fe1e8654505c9838eed470312fde1bf72f423b1e9725ec42 Nov 02 21:09:48.078025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.078000 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:09:48.741966 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.741920 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"3364f459642588698de26b6e88af5235ff3318ac0d5d5c5f189b70ecf54a7cb7"} Nov 02 21:09:48.742122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.742007 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"0758eb679134e10ba9a5a41df88444b492276ca5c1554e771a43bee50b9f49d9"} Nov 02 21:09:48.742122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.742044 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"50757a49b90c235e4ebb052957b983e4daa6c79ffa34e13ed6288f9f6d6794fd"} Nov 02 21:09:48.742122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.742078 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"82f4c364a96c921ebe68435bb30d4f9ff5eeb208645174dbf712ff8c22115a8c"} Nov 02 21:09:48.742122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.742105 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"17f74ff8141c91e5f7d2cf5172f092d92b6eea97c894ab4deef45ce403b27084"} Nov 02 21:09:48.742294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.742139 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbe35f1-c94d-4036-b80f-449bf85d5b7a","Type":"ContainerStarted","Data":"b1f505aa6b6ca140e2bb76b26639b744bf2cadcc1d00fd94eb51031aa2ab43d7"} Nov 02 21:09:48.743485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.743462 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" event={"ID":"53127e36-620e-41bb-940f-e4548cebc2ee","Type":"ContainerStarted","Data":"6ed304042c9800f8fe1e8654505c9838eed470312fde1bf72f423b1e9725ec42"} Nov 02 21:09:48.857249 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:48.857209 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.840289961 podStartE2EDuration="3.857196131s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.90617ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:48.856757346 +0000 UTC m=+28.405007154" watchObservedRunningTime="2026-11-02 21:09:48.857196131 +0000 UTC m=+28.405445902" Nov 02 21:09:49.224843 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.224816 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:49.225212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225161 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="prometheus" containerID="cri-o://a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" gracePeriod=600 Nov 02 21:09:49.225252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225197 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="config-reloader" containerID="cri-o://b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" gracePeriod=600 Nov 02 21:09:49.225283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225237 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy" containerID="cri-o://98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" gracePeriod=600 Nov 02 21:09:49.225368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225345 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" gracePeriod=600 Nov 02 21:09:49.225468 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225383 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="thanos-sidecar" containerID="cri-o://880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" gracePeriod=600 Nov 02 21:09:49.225539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.225455 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-web" containerID="cri-o://7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" gracePeriod=600 Nov 02 21:09:49.524201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.524174 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:49.638325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638301 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638452 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638340 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638452 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638391 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638452 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638421 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638450 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638488 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638519 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638568 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638606 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638640 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638667 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638700 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638734 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638767 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.638835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638806 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638835 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638864 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638902 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638930 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.638988 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") pod \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\" (UID: \"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8\") " Nov 02 21:09:49.639140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639082 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.639425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639288 5859 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.639425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639348 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.639425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639379 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.639654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639628 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.640313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.639991 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.641607 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.640851 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:49.641607 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.641487 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.642045 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.641847 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.643177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.643138 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.643371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.643342 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:49.643807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.643783 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:49.644085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.644042 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.644471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.644141 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.644560 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.644542 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "kube-api-access-j8ws4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:49.644903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.644715 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.645128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.645029 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.645409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.645371 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:49.646708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.646682 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.647244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.646923 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.658289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.658253 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config" pod "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" (UID: "823cc8d3-d22a-464b-b02a-d84d6b2e4cd8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:49.740714 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740683 5859 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-metrics-client-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740718 5859 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740734 5859 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740747 5859 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-grpc-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740761 5859 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740776 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ws4\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-kube-api-access-j8ws4\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740790 5859 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-kube-rbac-proxy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740805 5859 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740819 5859 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-tls-assets\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740835 5859 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.740848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740849 5859 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740867 5859 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740882 5859 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740897 5859 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740940 5859 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-web-config\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740981 5859 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-metrics-client-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.740990 5859 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-config-out\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.741000 5859 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-prometheus-k8s-db\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.741363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.741009 5859 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753556 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" exitCode=0 Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753590 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" exitCode=0 Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753599 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" exitCode=0 Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753608 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" exitCode=0 Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753616 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" exitCode=0 Nov 02 21:09:49.753629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753623 5859 generic.go:356] "Generic (PLEG): container finished" podID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" exitCode=0 Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753737 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753808 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753884 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753931 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.753999 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.754033 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.754043 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} Nov 02 21:09:49.754095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.754087 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} Nov 02 21:09:49.754422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.754124 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"823cc8d3-d22a-464b-b02a-d84d6b2e4cd8","Type":"ContainerDied","Data":"c97dc72bc48f9fc99f34fe08c9a6e068300d004de941adce452592e4a4026bad"} Nov 02 21:09:49.763023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.763005 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.772850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.772833 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.781179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.781145 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.789441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.789363 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.798083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.798067 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.806979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.806940 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.813646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.813631 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.813893 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.813877 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.813945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.813901 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.813945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.813930 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.814158 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.814143 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.814194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814164 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.814194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814177 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.814353 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.814338 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.814387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814356 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.814387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814368 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.814586 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.814572 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.814626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814590 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.814626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814603 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.814784 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.814770 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.814834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814786 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.814834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814796 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.814984 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.814967 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.815075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814983 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.815075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.814996 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.815236 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:49.815211 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.815273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815248 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.815273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815265 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.815482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815468 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.815517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815483 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.815663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815648 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.815703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815663 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.815847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815829 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.815847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.815846 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.816056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816041 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.816110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816055 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.816246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816231 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.816290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816246 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.816420 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816405 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.816466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816419 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.816606 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816591 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.816606 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816605 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.816787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816771 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.816833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816798 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.817014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.816999 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.817014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817013 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.817215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817200 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.817262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817216 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.817410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817395 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.817445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817410 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.817610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817593 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.817644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817611 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.817828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817813 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.817865 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.817828 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.818032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818016 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.818078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818032 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.818237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818222 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.818273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818237 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.818412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818395 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.818412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818408 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.818586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818561 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.818586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818584 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.818781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818764 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.818816 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818782 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.818986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818972 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.818986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.818985 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.819200 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819183 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.819243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819201 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.819423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819405 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.819462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819424 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.819621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819602 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.819659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819621 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.819821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819808 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.819857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.819821 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.820023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820007 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.820058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820024 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.820241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820227 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.820241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820241 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.820426 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820409 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.820463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820428 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.820613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820590 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.820613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820612 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.820807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820792 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.820807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820806 5859 scope.go:122] "RemoveContainer" containerID="d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c" Nov 02 21:09:49.821010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.820994 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c"} err="failed to get container status \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": rpc error: code = NotFound desc = could not find container \"d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c\": container with ID starting with d795dbff1fb4b4e7966d2a80d3bd0cec9914c7b798fe82c09ed6570bfbc2726c not found: ID does not exist" Nov 02 21:09:49.821075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821009 5859 scope.go:122] "RemoveContainer" containerID="98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47" Nov 02 21:09:49.821180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821167 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47"} err="failed to get container status \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": rpc error: code = NotFound desc = could not find container \"98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47\": container with ID starting with 98e29645235b67bb3ef8b2ca041fe8037cb86cbe5351469e22cf755d46f6be47 not found: ID does not exist" Nov 02 21:09:49.821220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821181 5859 scope.go:122] "RemoveContainer" containerID="7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d" Nov 02 21:09:49.821382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821365 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d"} err="failed to get container status \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": rpc error: code = NotFound desc = could not find container \"7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d\": container with ID starting with 7c9a3ab7109ded6b9191a676354e68e81534bdfacb73dcab8a97386db5a0399d not found: ID does not exist" Nov 02 21:09:49.821382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821381 5859 scope.go:122] "RemoveContainer" containerID="880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f" Nov 02 21:09:49.821586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821567 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f"} err="failed to get container status \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": rpc error: code = NotFound desc = could not find container \"880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f\": container with ID starting with 880cf3a65ece3a89c6ac36fe5cb23d04cc3cc7504a7af2ba5234fffcfb60590f not found: ID does not exist" Nov 02 21:09:49.821645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821585 5859 scope.go:122] "RemoveContainer" containerID="b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6" Nov 02 21:09:49.821787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821768 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6"} err="failed to get container status \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": rpc error: code = NotFound desc = could not find container \"b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6\": container with ID starting with b7e5349cb31782795460782445bd059ae7ba37d3fe2605dc32327133ee917be6 not found: ID does not exist" Nov 02 21:09:49.821828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.821787 5859 scope.go:122] "RemoveContainer" containerID="a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca" Nov 02 21:09:49.822048 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.822033 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca"} err="failed to get container status \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": rpc error: code = NotFound desc = could not find container \"a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca\": container with ID starting with a6fd819361d0a88bff4c98d4e4c889d7f6fa5f72d77d56ecaa8d140a28285dca not found: ID does not exist" Nov 02 21:09:49.822095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.822048 5859 scope.go:122] "RemoveContainer" containerID="5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6" Nov 02 21:09:49.822241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.822227 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6"} err="failed to get container status \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": rpc error: code = NotFound desc = could not find container \"5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6\": container with ID starting with 5907876f4c114eba488433c6501234f5cbdecade97942a7a7b08412ff68fa4a6 not found: ID does not exist" Nov 02 21:09:49.917646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.917612 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:49.924132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.924105 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:49.953860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.953835 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:49.954187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954169 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-thanos" Nov 02 21:09:49.954187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954186 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-thanos" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954199 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954208 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954223 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="thanos-sidecar" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954229 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="thanos-sidecar" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954244 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-web" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954252 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-web" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954283 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="init-config-reloader" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954290 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="init-config-reloader" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954305 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="prometheus" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954311 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="prometheus" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954329 5859 cpu_manager.go:414] "containerMap: removing container" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="config-reloader" Nov 02 21:09:49.954345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954337 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="config-reloader" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954397 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954406 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="thanos-sidecar" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954413 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-web" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954420 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="init-config-reloader" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954426 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="prometheus" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954432 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="config-reloader" Nov 02 21:09:49.954794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.954439 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" containerName="kube-rbac-proxy-thanos" Nov 02 21:09:49.971878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:49.971853 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.022369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022297 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:09:50.022369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022307 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:09:50.022369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022354 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022392 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022395 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022401 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022422 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022455 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:09:50.022616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.022468 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:09:50.023073 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.023046 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ptqqmo5n63c4\"" Nov 02 21:09:50.023365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.023345 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:09:50.023448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.023434 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:09:50.023526 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.023508 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:09:50.023645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.023567 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sb2rf\"" Nov 02 21:09:50.043270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043239 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043291 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043379 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043420 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043445 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043538 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043592 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043621 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043647 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043672 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043719 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5sxb\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-kube-api-access-d5sxb\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043771 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043809 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.043971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.043853 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044005 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044035 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044086 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044118 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044137 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.044223 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.044170 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.069711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.069663 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:09:50.069711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.069691 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:09:50.070635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.070612 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:50.145001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.144974 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.145163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145023 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145486 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145534 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145582 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145616 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145641 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145670 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145672 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145695 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145719 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sxb\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-kube-api-access-d5sxb\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145761 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145799 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145820 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145856 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145884 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145909 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145939 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.145987 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.146033 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.146138 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.146396 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.147146 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.147213 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.149418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.148590 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.150779 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.149491 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.150779 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.150485 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.150779 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.150490 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.151552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.151514 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc420536-56b5-400c-8d7a-fcefa442eaec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.152789 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.152771 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.152891 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.152844 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.158935 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc420536-56b5-400c-8d7a-fcefa442eaec-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.159002 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.159090 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.159106 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.159116 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.159277 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.159152 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-config\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.160314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.160298 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc420536-56b5-400c-8d7a-fcefa442eaec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.201598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.201578 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sxb\" (UniqueName: \"kubernetes.io/projected/fc420536-56b5-400c-8d7a-fcefa442eaec-kube-api-access-d5sxb\") pod \"prometheus-k8s-0\" (UID: \"fc420536-56b5-400c-8d7a-fcefa442eaec\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.377984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.377939 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:50.568790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.568765 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823cc8d3-d22a-464b-b02a-d84d6b2e4cd8" path="/var/lib/kubelet/pods/823cc8d3-d22a-464b-b02a-d84d6b2e4cd8/volumes" Nov 02 21:09:50.871590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:50.871566 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:09:50.878821 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:09:50.878791 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc420536_56b5_400c_8d7a_fcefa442eaec.slice/crio-c2ab0109145ae46be9b29f6b809cce2ec62f708f3825e40e61b21ab424a6150b WatchSource:0}: Error finding container c2ab0109145ae46be9b29f6b809cce2ec62f708f3825e40e61b21ab424a6150b: Status 404 returned error can't find the container with id c2ab0109145ae46be9b29f6b809cce2ec62f708f3825e40e61b21ab424a6150b Nov 02 21:09:51.763890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.763859 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" event={"ID":"53127e36-620e-41bb-940f-e4548cebc2ee","Type":"ContainerStarted","Data":"b430a29fe27ad925098926d1b06012215b75dfcf2ffdd674b87601428a53ad3c"} Nov 02 21:09:51.764383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.763916 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" event={"ID":"53127e36-620e-41bb-940f-e4548cebc2ee","Type":"ContainerStarted","Data":"0b8b51cb927cb574793112b978cf39c917f51f2e8556897bd0b5682433f86ce2"} Nov 02 21:09:51.764383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.763946 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" event={"ID":"53127e36-620e-41bb-940f-e4548cebc2ee","Type":"ContainerStarted","Data":"41ebab2aeafa96cc38fe7631635ab0a65b3e6685cf670117eb6b96646c993e49"} Nov 02 21:09:51.765514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.765491 5859 generic.go:356] "Generic (PLEG): container finished" podID="fc420536-56b5-400c-8d7a-fcefa442eaec" containerID="dcb72023f3490f362bdd2dfadc15e34d93b87da56b618148f63b75c31fe14b16" exitCode=0 Nov 02 21:09:51.765633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.765521 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerDied","Data":"dcb72023f3490f362bdd2dfadc15e34d93b87da56b618148f63b75c31fe14b16"} Nov 02 21:09:51.765633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.765576 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"c2ab0109145ae46be9b29f6b809cce2ec62f708f3825e40e61b21ab424a6150b"} Nov 02 21:09:51.874690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:51.874634 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8589895978-xdksc" podStartSLOduration=2.164098975 podStartE2EDuration="4.874616107s" totalImagesPullingTime="2.710517132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:51.873308337 +0000 UTC m=+31.421558155" watchObservedRunningTime="2026-11-02 21:09:51.874616107 +0000 UTC m=+31.422865888" Nov 02 21:09:52.775770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775726 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"7602f889fcb4e111ae75c435772d1ca3df5e10513e74a2e77c43b7b873976922"} Nov 02 21:09:52.776148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775818 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"c42e8e4fbd70aa27aaa6b4008368360617fc1a4ec8e395fa44eb5158fdd63cbf"} Nov 02 21:09:52.776148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775859 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"1ccfb1c0eca0b7156761dc5c09078c4cdfe63f8f9b05e52a38daa9968d86fcb5"} Nov 02 21:09:52.776148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775888 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"a0dd75b019c8877ddcb41a018efe31318c153c03febf7779a93def5dbd7f3224"} Nov 02 21:09:52.776148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775910 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"9980648056f9f731a5bb51184f6d3791cb62678e6662ccb51d45394fe56c6302"} Nov 02 21:09:52.776148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.775944 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc420536-56b5-400c-8d7a-fcefa442eaec","Type":"ContainerStarted","Data":"a3b8c87ee1082507820f91856cb5117f8b4bfbdee506e58c9faf9e5fbad5f233"} Nov 02 21:09:52.903361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.903319 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.883433812 podStartE2EDuration="3.903306225s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.872413ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:52.90244429 +0000 UTC m=+32.450694090" watchObservedRunningTime="2026-11-02 21:09:52.903306225 +0000 UTC m=+32.451556005" Nov 02 21:09:52.943352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:52.943332 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pt5zs" Nov 02 21:09:55.378182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:09:55.378150 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:09:55.942600 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:09:55.942569 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:10:25.942679 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:10:25.942641 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:10:33.396647 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:10:33.396615 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:10:33.411476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:10:33.411455 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:10:33.933012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:10:33.932987 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:10:55.944405 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:10:55.944364 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:11:25.943779 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:11:25.943699 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:11:55.942658 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:11:55.942616 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:12:25.944653 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:12:25.944614 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:12:55.944425 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:12:55.944386 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:12:58.106778 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.106751 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9"] Nov 02 21:12:58.110417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.110402 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.131666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.131638 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws"] Nov 02 21:12:58.134930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.134914 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.159464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.159447 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:12:58.177751 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.177732 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pd9kp\"" Nov 02 21:12:58.178384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.178364 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k"] Nov 02 21:12:58.181634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.181616 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.206906 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.206884 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9"] Nov 02 21:12:58.211250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.211230 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws"] Nov 02 21:12:58.211736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.211721 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz"] Nov 02 21:12:58.214987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.214972 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.239001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.238980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.239092 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239006 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.239092 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239038 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.239092 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239059 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmthn\" (UniqueName: \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.239206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239172 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxtx\" (UniqueName: \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.239206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239195 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.239286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239270 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.239348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239331 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc49\" (UniqueName: \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.239396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.239361 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.254266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.254243 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k"] Nov 02 21:12:58.259792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.259776 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:12:58.264108 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.264087 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz"] Nov 02 21:12:58.266424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.266397 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-9n7nj\"" Nov 02 21:12:58.266528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.266444 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2t6z9\"" Nov 02 21:12:58.266528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.266524 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-t5w82\"" Nov 02 21:12:58.340225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340204 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.340322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340239 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.340322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340268 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmthn\" (UniqueName: \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.340416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340321 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.340416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340365 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxtx\" (UniqueName: \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.340416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340383 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.340631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340451 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.340631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340490 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhv9\" (UniqueName: \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.340631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340545 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc49\" (UniqueName: \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.340631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340571 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.340912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340643 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.340912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340681 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.340912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340726 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.340912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340726 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.340912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340825 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.341279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.340931 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.341279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.341019 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.341279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.341044 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.394561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.394496 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmthn\" (UniqueName: \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.394561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.394515 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc49\" (UniqueName: \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.394853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.394836 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxtx\" (UniqueName: \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.425674 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.425651 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:12:58.442084 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.442061 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhv9\" (UniqueName: \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.442194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.442166 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.442251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.442198 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.442526 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.442507 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.442612 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.442547 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.449821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.449801 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:12:58.496241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.496214 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhv9\" (UniqueName: \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.496998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.496979 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:12:58.529262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.529232 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:12:58.825147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.825119 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9"] Nov 02 21:12:58.831369 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:12:58.831347 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f89fe1_584e_49f9_9061_14167d508370.slice/crio-4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650 WatchSource:0}: Error finding container 4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650: Status 404 returned error can't find the container with id 4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650 Nov 02 21:12:58.840409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.840387 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws"] Nov 02 21:12:58.847089 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:12:58.847032 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2bbe3b_f602_41a0_8051_6f292314d852.slice/crio-2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c WatchSource:0}: Error finding container 2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c: Status 404 returned error can't find the container with id 2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c Nov 02 21:12:58.878467 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.878447 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k"] Nov 02 21:12:58.884086 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:12:58.884065 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be05a69_3525_4016_ad26_1758266fd5c4.slice/crio-ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5 WatchSource:0}: Error finding container ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5: Status 404 returned error can't find the container with id ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5 Nov 02 21:12:58.899857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:58.899838 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz"] Nov 02 21:12:58.907144 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:12:58.907125 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4775284_0b69_42ed_a38d_74c57de9b712.slice/crio-422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3 WatchSource:0}: Error finding container 422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3: Status 404 returned error can't find the container with id 422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3 Nov 02 21:12:59.438037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:59.438006 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerStarted","Data":"4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650"} Nov 02 21:12:59.439104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:59.439080 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerStarted","Data":"2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c"} Nov 02 21:12:59.440354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:59.440329 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerStarted","Data":"422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3"} Nov 02 21:12:59.441414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:12:59.441395 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" event={"ID":"5be05a69-3525-4016-ad26-1758266fd5c4","Type":"ContainerStarted","Data":"ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5"} Nov 02 21:13:08.493248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.493216 5859 generic.go:356] "Generic (PLEG): container finished" podID="31f89fe1-584e-49f9-9061-14167d508370" containerID="0b417959da7708c1865a17a009fe076b6057bf655eb5078af83f1f108cf14204" exitCode=0 Nov 02 21:13:08.493611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.493297 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerDied","Data":"0b417959da7708c1865a17a009fe076b6057bf655eb5078af83f1f108cf14204"} Nov 02 21:13:08.494914 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.494889 5859 generic.go:356] "Generic (PLEG): container finished" podID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerID="36a0dfc730615cbd5eccfc8d93cefe70c43a00f237810a0cecf4b413e5354962" exitCode=0 Nov 02 21:13:08.495030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.494986 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerDied","Data":"36a0dfc730615cbd5eccfc8d93cefe70c43a00f237810a0cecf4b413e5354962"} Nov 02 21:13:08.496822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.496799 5859 generic.go:356] "Generic (PLEG): container finished" podID="b4775284-0b69-42ed-a38d-74c57de9b712" containerID="449f88c2e02fccabe16d73604ec8e1415c1a88103f106455e97b890899ffaa6d" exitCode=0 Nov 02 21:13:08.496924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.496876 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerDied","Data":"449f88c2e02fccabe16d73604ec8e1415c1a88103f106455e97b890899ffaa6d"} Nov 02 21:13:08.498387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.498363 5859 generic.go:356] "Generic (PLEG): container finished" podID="5be05a69-3525-4016-ad26-1758266fd5c4" containerID="612cfecef1d164edaaba5c155a0104769c7f231cbbe4243e944a399a45b39989" exitCode=0 Nov 02 21:13:08.498488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:08.498420 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" event={"ID":"5be05a69-3525-4016-ad26-1758266fd5c4","Type":"ContainerDied","Data":"612cfecef1d164edaaba5c155a0104769c7f231cbbe4243e944a399a45b39989"} Nov 02 21:13:10.507558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:10.507536 5859 generic.go:356] "Generic (PLEG): container finished" podID="5be05a69-3525-4016-ad26-1758266fd5c4" containerID="53d16206b2853183819788a89e9309241ccdf1d49156dd0217c4b0c859683444" exitCode=0 Nov 02 21:13:10.507866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:10.507585 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" event={"ID":"5be05a69-3525-4016-ad26-1758266fd5c4","Type":"ContainerDied","Data":"53d16206b2853183819788a89e9309241ccdf1d49156dd0217c4b0c859683444"} Nov 02 21:13:11.513939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.513865 5859 generic.go:356] "Generic (PLEG): container finished" podID="31f89fe1-584e-49f9-9061-14167d508370" containerID="8af1aba353f566e15f7f8cf6213ac7219b4eaa6626403f08ae1444d3e2d0089f" exitCode=0 Nov 02 21:13:11.514330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.513935 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerDied","Data":"8af1aba353f566e15f7f8cf6213ac7219b4eaa6626403f08ae1444d3e2d0089f"} Nov 02 21:13:11.515836 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.515813 5859 generic.go:356] "Generic (PLEG): container finished" podID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerID="591558f9b49ea4e99b01dc12015f0b7a413026b380a805dd151f3a429c20a857" exitCode=0 Nov 02 21:13:11.515925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.515901 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerDied","Data":"591558f9b49ea4e99b01dc12015f0b7a413026b380a805dd151f3a429c20a857"} Nov 02 21:13:11.520534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.520509 5859 generic.go:356] "Generic (PLEG): container finished" podID="b4775284-0b69-42ed-a38d-74c57de9b712" containerID="63c99bc5e771f8e0c1a12cb09c3eaf92f72e004320c1e818e5cc7339d7415116" exitCode=0 Nov 02 21:13:11.520619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:11.520566 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerDied","Data":"63c99bc5e771f8e0c1a12cb09c3eaf92f72e004320c1e818e5cc7339d7415116"} Nov 02 21:13:20.573752 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:20.573716 5859 generic.go:356] "Generic (PLEG): container finished" podID="5be05a69-3525-4016-ad26-1758266fd5c4" containerID="a885bf38331e1bff61a9ffe87db141bc985a2b7b7fd92aa6853c40c52b802cf7" exitCode=0 Nov 02 21:13:20.574217 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:20.573804 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" event={"ID":"5be05a69-3525-4016-ad26-1758266fd5c4","Type":"ContainerDied","Data":"a885bf38331e1bff61a9ffe87db141bc985a2b7b7fd92aa6853c40c52b802cf7"} Nov 02 21:13:21.585287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.585248 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerStarted","Data":"5175934d11235bcf435322da4a40f6abd24e0c6b2e51101381c89e20a445f0ff"} Nov 02 21:13:21.587510 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.587485 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerStarted","Data":"a096b0840fb428aa6e60f220cdc556ec3446e9f0d0676554825ee8a0baac2d18"} Nov 02 21:13:21.589491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.589469 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerStarted","Data":"c981bb7ac8d176fdd19dcbbb5b711d7c8d64502b50ca139651e8d2de739f6e77"} Nov 02 21:13:21.711289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.711267 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:13:21.726497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.726436 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" podStartSLOduration=2.428259119 podStartE2EDuration="23.726418767s" totalImagesPullingTime="21.298159648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:21.72560783 +0000 UTC m=+241.273857630" watchObservedRunningTime="2026-11-02 21:13:21.726418767 +0000 UTC m=+241.274668550" Nov 02 21:13:21.765755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.765733 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\" (UniqueName: \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\") pod \"5be05a69-3525-4016-ad26-1758266fd5c4\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " Nov 02 21:13:21.765884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.765789 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\") pod \"5be05a69-3525-4016-ad26-1758266fd5c4\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " Nov 02 21:13:21.765884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.765820 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\") pod \"5be05a69-3525-4016-ad26-1758266fd5c4\" (UID: \"5be05a69-3525-4016-ad26-1758266fd5c4\") " Nov 02 21:13:21.766298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.766271 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle" pod "5be05a69-3525-4016-ad26-1758266fd5c4" (UID: "5be05a69-3525-4016-ad26-1758266fd5c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:21.767742 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.767720 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn" pod "5be05a69-3525-4016-ad26-1758266fd5c4" (UID: "5be05a69-3525-4016-ad26-1758266fd5c4"). InnerVolumeSpecName "kube-api-access-pmthn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:21.770528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.770510 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util" pod "5be05a69-3525-4016-ad26-1758266fd5c4" (UID: "5be05a69-3525-4016-ad26-1758266fd5c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:21.833613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.833574 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" podStartSLOduration=2.903632284 podStartE2EDuration="23.83356171s" totalImagesPullingTime="20.929929426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:21.832183978 +0000 UTC m=+241.380433780" watchObservedRunningTime="2026-11-02 21:13:21.83356171 +0000 UTC m=+241.381811493" Nov 02 21:13:21.867266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.867245 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmthn\" (UniqueName: \"kubernetes.io/projected/5be05a69-3525-4016-ad26-1758266fd5c4-kube-api-access-pmthn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:21.867266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.867265 5859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:21.867384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:21.867275 5859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5be05a69-3525-4016-ad26-1758266fd5c4-util\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:22.051175 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.051129 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" podStartSLOduration=2.834545143 podStartE2EDuration="24.051115097s" totalImagesPullingTime="21.216569954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:22.050037954 +0000 UTC m=+241.598287745" watchObservedRunningTime="2026-11-02 21:13:22.051115097 +0000 UTC m=+241.599364877" Nov 02 21:13:22.594810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.594780 5859 generic.go:356] "Generic (PLEG): container finished" podID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerID="a096b0840fb428aa6e60f220cdc556ec3446e9f0d0676554825ee8a0baac2d18" exitCode=0 Nov 02 21:13:22.595199 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.594863 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerDied","Data":"a096b0840fb428aa6e60f220cdc556ec3446e9f0d0676554825ee8a0baac2d18"} Nov 02 21:13:22.597043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.597014 5859 generic.go:356] "Generic (PLEG): container finished" podID="b4775284-0b69-42ed-a38d-74c57de9b712" containerID="c981bb7ac8d176fdd19dcbbb5b711d7c8d64502b50ca139651e8d2de739f6e77" exitCode=0 Nov 02 21:13:22.597159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.597085 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerDied","Data":"c981bb7ac8d176fdd19dcbbb5b711d7c8d64502b50ca139651e8d2de739f6e77"} Nov 02 21:13:22.598879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.598861 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" Nov 02 21:13:22.598991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.598891 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9s7k" event={"ID":"5be05a69-3525-4016-ad26-1758266fd5c4","Type":"ContainerDied","Data":"ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5"} Nov 02 21:13:22.598991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.598927 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2c35d756e011d9b9ca089872b318a13a1119e8b33c51e5f5bf3a9c7512c3e5" Nov 02 21:13:22.601118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.601100 5859 generic.go:356] "Generic (PLEG): container finished" podID="31f89fe1-584e-49f9-9061-14167d508370" containerID="5175934d11235bcf435322da4a40f6abd24e0c6b2e51101381c89e20a445f0ff" exitCode=0 Nov 02 21:13:22.601216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:22.601128 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerDied","Data":"5175934d11235bcf435322da4a40f6abd24e0c6b2e51101381c89e20a445f0ff"} Nov 02 21:13:23.735935 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.735913 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:13:23.776773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.776747 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:13:23.779563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.779547 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:13:23.782880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.782865 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\" (UniqueName: \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\") pod \"31f89fe1-584e-49f9-9061-14167d508370\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " Nov 02 21:13:23.782926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.782905 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\") pod \"31f89fe1-584e-49f9-9061-14167d508370\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " Nov 02 21:13:23.783010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.782999 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\") pod \"31f89fe1-584e-49f9-9061-14167d508370\" (UID: \"31f89fe1-584e-49f9-9061-14167d508370\") " Nov 02 21:13:23.783379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.783357 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle" pod "31f89fe1-584e-49f9-9061-14167d508370" (UID: "31f89fe1-584e-49f9-9061-14167d508370"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.784753 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.784733 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx" pod "31f89fe1-584e-49f9-9061-14167d508370" (UID: "31f89fe1-584e-49f9-9061-14167d508370"). InnerVolumeSpecName "kube-api-access-ksxtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:23.788485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.788464 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util" pod "31f89fe1-584e-49f9-9061-14167d508370" (UID: "31f89fe1-584e-49f9-9061-14167d508370"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.884023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.883968 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\") pod \"1b2bbe3b-f602-41a0-8051-6f292314d852\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " Nov 02 21:13:23.884023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884012 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\") pod \"b4775284-0b69-42ed-a38d-74c57de9b712\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " Nov 02 21:13:23.884171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884038 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\") pod \"b4775284-0b69-42ed-a38d-74c57de9b712\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " Nov 02 21:13:23.884171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884062 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\" (UniqueName: \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\") pod \"1b2bbe3b-f602-41a0-8051-6f292314d852\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " Nov 02 21:13:23.884171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884079 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\") pod \"1b2bbe3b-f602-41a0-8051-6f292314d852\" (UID: \"1b2bbe3b-f602-41a0-8051-6f292314d852\") " Nov 02 21:13:23.884171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884111 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\" (UniqueName: \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\") pod \"b4775284-0b69-42ed-a38d-74c57de9b712\" (UID: \"b4775284-0b69-42ed-a38d-74c57de9b712\") " Nov 02 21:13:23.884375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884349 5859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.884375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884367 5859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31f89fe1-584e-49f9-9061-14167d508370-util\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.884477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884382 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxtx\" (UniqueName: \"kubernetes.io/projected/31f89fe1-584e-49f9-9061-14167d508370-kube-api-access-ksxtx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.884587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884566 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle" pod "1b2bbe3b-f602-41a0-8051-6f292314d852" (UID: "1b2bbe3b-f602-41a0-8051-6f292314d852"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.884743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.884720 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle" pod "b4775284-0b69-42ed-a38d-74c57de9b712" (UID: "b4775284-0b69-42ed-a38d-74c57de9b712"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.885987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.885963 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49" pod "1b2bbe3b-f602-41a0-8051-6f292314d852" (UID: "1b2bbe3b-f602-41a0-8051-6f292314d852"). InnerVolumeSpecName "kube-api-access-fdc49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:23.886430 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.886417 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9" pod "b4775284-0b69-42ed-a38d-74c57de9b712" (UID: "b4775284-0b69-42ed-a38d-74c57de9b712"). InnerVolumeSpecName "kube-api-access-pbhv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:23.888454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.888437 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util" pod "b4775284-0b69-42ed-a38d-74c57de9b712" (UID: "b4775284-0b69-42ed-a38d-74c57de9b712"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.889993 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.889975 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util" pod "1b2bbe3b-f602-41a0-8051-6f292314d852" (UID: "1b2bbe3b-f602-41a0-8051-6f292314d852"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:23.985401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985379 5859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.985401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985397 5859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4775284-0b69-42ed-a38d-74c57de9b712-util\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.985537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985408 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc49\" (UniqueName: \"kubernetes.io/projected/1b2bbe3b-f602-41a0-8051-6f292314d852-kube-api-access-fdc49\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.985537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985417 5859 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-bundle\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.985537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985426 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhv9\" (UniqueName: \"kubernetes.io/projected/b4775284-0b69-42ed-a38d-74c57de9b712-kube-api-access-pbhv9\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:23.985537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:23.985435 5859 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b2bbe3b-f602-41a0-8051-6f292314d852-util\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:24.610247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.610226 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" Nov 02 21:13:24.610367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.610233 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlv9" event={"ID":"31f89fe1-584e-49f9-9061-14167d508370","Type":"ContainerDied","Data":"4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650"} Nov 02 21:13:24.610367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.610286 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd9b932dd1689ae1d9f6cb2b31c56c91e5a9842d5912679966fe42e0cde6650" Nov 02 21:13:24.612493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.612471 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" Nov 02 21:13:24.612600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.612471 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wsjws" event={"ID":"1b2bbe3b-f602-41a0-8051-6f292314d852","Type":"ContainerDied","Data":"2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c"} Nov 02 21:13:24.612659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.612599 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5664dc4db68c0b9bc615430fefa90113f8d944236146485f2e7ff62f95005c" Nov 02 21:13:24.614596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.614575 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" event={"ID":"b4775284-0b69-42ed-a38d-74c57de9b712","Type":"ContainerDied","Data":"422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3"} Nov 02 21:13:24.614698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.614599 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847annplz" Nov 02 21:13:24.614698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:24.614613 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422f6bcb55e8286c6b3ff92330728581bdc54a88fbc3e7236149e1374003f0e3" Nov 02 21:13:25.941552 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:13:25.941522 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:13:40.166665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166631 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td"] Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166933 5859 cpu_manager.go:414] "containerMap: removing container" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166943 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166965 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166970 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166980 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166985 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166992 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.166996 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167004 5859 cpu_manager.go:414] "containerMap: removing container" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167009 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167015 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167019 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167024 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167029 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167036 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167041 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167046 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167050 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167058 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167062 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="util" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167068 5859 cpu_manager.go:414] "containerMap: removing container" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167072 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167084 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167088 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="pull" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167131 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be05a69-3525-4016-ad26-1758266fd5c4" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167138 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4775284-0b69-42ed-a38d-74c57de9b712" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167143 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2bbe3b-f602-41a0-8051-6f292314d852" containerName="extract" Nov 02 21:13:40.167157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.167149 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f89fe1-584e-49f9-9061-14167d508370" containerName="extract" Nov 02 21:13:40.170080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.170063 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.218902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.218879 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-d66sx\"" Nov 02 21:13:40.219023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.218907 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 21:13:40.219023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.218929 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:13:40.266496 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.266473 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td"] Nov 02 21:13:40.301014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.300989 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1545eb9-1f57-4142-ac92-a8b805570dc7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.301109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.301020 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n2s\" (UniqueName: \"kubernetes.io/projected/e1545eb9-1f57-4142-ac92-a8b805570dc7-kube-api-access-r2n2s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.401394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.401373 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1545eb9-1f57-4142-ac92-a8b805570dc7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.401498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.401405 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n2s\" (UniqueName: \"kubernetes.io/projected/e1545eb9-1f57-4142-ac92-a8b805570dc7-kube-api-access-r2n2s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.401728 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.401712 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1545eb9-1f57-4142-ac92-a8b805570dc7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.464578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.464519 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n2s\" (UniqueName: \"kubernetes.io/projected/e1545eb9-1f57-4142-ac92-a8b805570dc7-kube-api-access-r2n2s\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5s7td\" (UID: \"e1545eb9-1f57-4142-ac92-a8b805570dc7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.481451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.481433 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" Nov 02 21:13:40.952104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:40.952074 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td"] Nov 02 21:13:40.959333 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:40.959306 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1545eb9_1f57_4142_ac92_a8b805570dc7.slice/crio-e70cadde0b522f5876103a6ecaf39771764d021f57e4432c47475c7b3c99bbc9 WatchSource:0}: Error finding container e70cadde0b522f5876103a6ecaf39771764d021f57e4432c47475c7b3c99bbc9: Status 404 returned error can't find the container with id e70cadde0b522f5876103a6ecaf39771764d021f57e4432c47475c7b3c99bbc9 Nov 02 21:13:41.675048 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:41.675015 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" event={"ID":"e1545eb9-1f57-4142-ac92-a8b805570dc7","Type":"ContainerStarted","Data":"e70cadde0b522f5876103a6ecaf39771764d021f57e4432c47475c7b3c99bbc9"} Nov 02 21:13:43.774903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.774869 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4"] Nov 02 21:13:43.778381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.778363 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:43.828293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.828264 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrqg\" (UniqueName: \"kubernetes.io/projected/33a98961-7c4a-4fdd-a2d9-b1072164286d-kube-api-access-zvrqg\") pod \"dns-operator-controller-manager-8557cbdb84-vsgm4\" (UID: \"33a98961-7c4a-4fdd-a2d9-b1072164286d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:43.876013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.875392 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h7qrt\"" Nov 02 21:13:43.876013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.875392 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 21:13:43.876013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.875442 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:13:43.876013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.875474 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:13:43.877513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.877488 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4"] Nov 02 21:13:43.929412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.929385 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrqg\" (UniqueName: \"kubernetes.io/projected/33a98961-7c4a-4fdd-a2d9-b1072164286d-kube-api-access-zvrqg\") pod \"dns-operator-controller-manager-8557cbdb84-vsgm4\" (UID: \"33a98961-7c4a-4fdd-a2d9-b1072164286d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:43.983323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:43.983290 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrqg\" (UniqueName: \"kubernetes.io/projected/33a98961-7c4a-4fdd-a2d9-b1072164286d-kube-api-access-zvrqg\") pod \"dns-operator-controller-manager-8557cbdb84-vsgm4\" (UID: \"33a98961-7c4a-4fdd-a2d9-b1072164286d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:44.025394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.025332 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd"] Nov 02 21:13:44.033360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.033333 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.082872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.082842 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd"] Nov 02 21:13:44.085645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.085606 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 21:13:44.085795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.085721 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:13:44.085795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.085739 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-b85bh\"" Nov 02 21:13:44.092893 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.092873 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:44.130976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.130934 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69t4h\" (UniqueName: \"kubernetes.io/projected/92594e0e-348a-4926-8e16-a9115a077541-kube-api-access-69t4h\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.131112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.131059 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92594e0e-348a-4926-8e16-a9115a077541-tmp\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.231750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.231720 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92594e0e-348a-4926-8e16-a9115a077541-tmp\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.231899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.231803 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69t4h\" (UniqueName: \"kubernetes.io/projected/92594e0e-348a-4926-8e16-a9115a077541-kube-api-access-69t4h\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.232192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.232173 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92594e0e-348a-4926-8e16-a9115a077541-tmp\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.291442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.291385 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69t4h\" (UniqueName: \"kubernetes.io/projected/92594e0e-348a-4926-8e16-a9115a077541-kube-api-access-69t4h\") pod \"jobset-operator-674cd59dc5-tnxnd\" (UID: \"92594e0e-348a-4926-8e16-a9115a077541\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.358269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.358244 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" Nov 02 21:13:44.463264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.463235 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4"] Nov 02 21:13:44.470696 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:44.470663 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a98961_7c4a_4fdd_a2d9_b1072164286d.slice/crio-3350c7300ae7f92a244c0e2e1a768d447c9b5ab56b112373f4e3043cfe8975cc WatchSource:0}: Error finding container 3350c7300ae7f92a244c0e2e1a768d447c9b5ab56b112373f4e3043cfe8975cc: Status 404 returned error can't find the container with id 3350c7300ae7f92a244c0e2e1a768d447c9b5ab56b112373f4e3043cfe8975cc Nov 02 21:13:44.690353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.690313 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" event={"ID":"33a98961-7c4a-4fdd-a2d9-b1072164286d","Type":"ContainerStarted","Data":"3350c7300ae7f92a244c0e2e1a768d447c9b5ab56b112373f4e3043cfe8975cc"} Nov 02 21:13:44.731366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:44.731336 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd"] Nov 02 21:13:44.737991 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:44.737965 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92594e0e_348a_4926_8e16_a9115a077541.slice/crio-0d63682ad6e8326861859ebe22493b13c6104feac85ea62cff5043e36c6dd567 WatchSource:0}: Error finding container 0d63682ad6e8326861859ebe22493b13c6104feac85ea62cff5043e36c6dd567: Status 404 returned error can't find the container with id 0d63682ad6e8326861859ebe22493b13c6104feac85ea62cff5043e36c6dd567 Nov 02 21:13:45.696306 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:45.696269 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" event={"ID":"e1545eb9-1f57-4142-ac92-a8b805570dc7","Type":"ContainerStarted","Data":"ca627b331ada688a1f011532bb4ffa7571fa49f60d2fe58444945e2382fe218b"} Nov 02 21:13:45.697617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:45.697594 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" event={"ID":"92594e0e-348a-4926-8e16-a9115a077541","Type":"ContainerStarted","Data":"0d63682ad6e8326861859ebe22493b13c6104feac85ea62cff5043e36c6dd567"} Nov 02 21:13:45.899828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:45.899769 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5s7td" podStartSLOduration=1.842012684 podStartE2EDuration="5.899754526s" totalImagesPullingTime="4.057741842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:45.852231639 +0000 UTC m=+265.400481494" watchObservedRunningTime="2026-11-02 21:13:45.899754526 +0000 UTC m=+265.448004300" Nov 02 21:13:46.981782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:46.981751 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4"] Nov 02 21:13:46.993765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:46.993720 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.045291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.045203 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 21:13:47.047632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.045709 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 21:13:47.047632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.047502 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4"] Nov 02 21:13:47.047632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.047603 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pb7w5\"" Nov 02 21:13:47.058599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.058565 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlsv\" (UniqueName: \"kubernetes.io/projected/28db1cfd-ba37-4772-91fb-ad9f035e3624-kube-api-access-kqlsv\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.058716 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.058658 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28db1cfd-ba37-4772-91fb-ad9f035e3624-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.058774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.058730 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28db1cfd-ba37-4772-91fb-ad9f035e3624-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.159558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.159520 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlsv\" (UniqueName: \"kubernetes.io/projected/28db1cfd-ba37-4772-91fb-ad9f035e3624-kube-api-access-kqlsv\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.160991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.159621 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28db1cfd-ba37-4772-91fb-ad9f035e3624-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.160991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.159690 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28db1cfd-ba37-4772-91fb-ad9f035e3624-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.160991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.160924 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28db1cfd-ba37-4772-91fb-ad9f035e3624-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.180401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.180357 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28db1cfd-ba37-4772-91fb-ad9f035e3624-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.222788 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.222753 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlsv\" (UniqueName: \"kubernetes.io/projected/28db1cfd-ba37-4772-91fb-ad9f035e3624-kube-api-access-kqlsv\") pod \"kuadrant-console-plugin-66c4fb575d-qr5f4\" (UID: \"28db1cfd-ba37-4772-91fb-ad9f035e3624\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.308436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.308354 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" Nov 02 21:13:47.729797 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:47.729760 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4"] Nov 02 21:13:48.161573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.161540 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d"] Nov 02 21:13:48.172493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.172450 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.223621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.223539 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 21:13:48.224880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.224854 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qrm6q\"" Nov 02 21:13:48.225590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.225569 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d"] Nov 02 21:13:48.264833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.264801 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 21:13:48.269275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.269245 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4qf\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-kube-api-access-bf4qf\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.269441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.269305 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.370833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.370791 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.371249 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.371229 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4qf\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-kube-api-access-bf4qf\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.428006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.427903 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.428273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.428254 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4qf\" (UniqueName: \"kubernetes.io/projected/83465063-8244-4e60-8e40-9e2745717cc7-kube-api-access-bf4qf\") pod \"cert-manager-webhook-5dbbbf75cc-flz5d\" (UID: \"83465063-8244-4e60-8e40-9e2745717cc7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.489038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.489005 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:48.720140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.720025 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" event={"ID":"28db1cfd-ba37-4772-91fb-ad9f035e3624","Type":"ContainerStarted","Data":"b2ba12d03f0d05606a865b2665964c4fb551f346f8d0642551d5edad305e34c6"} Nov 02 21:13:48.872042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:48.872003 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d"] Nov 02 21:13:48.899846 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:48.899809 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83465063_8244_4e60_8e40_9e2745717cc7.slice/crio-03d92e6e5be8d3e30d8785ebe23c388125a2b3cd9b47aee87459137fa8a3f0c8 WatchSource:0}: Error finding container 03d92e6e5be8d3e30d8785ebe23c388125a2b3cd9b47aee87459137fa8a3f0c8: Status 404 returned error can't find the container with id 03d92e6e5be8d3e30d8785ebe23c388125a2b3cd9b47aee87459137fa8a3f0c8 Nov 02 21:13:49.726251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.726214 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" event={"ID":"92594e0e-348a-4926-8e16-a9115a077541","Type":"ContainerStarted","Data":"ae5ba821a30bceb4c224cc51e531db5b58ea83ce56aff43e38fa6d8f7d5a4167"} Nov 02 21:13:49.727466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.727441 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" event={"ID":"83465063-8244-4e60-8e40-9e2745717cc7","Type":"ContainerStarted","Data":"03d92e6e5be8d3e30d8785ebe23c388125a2b3cd9b47aee87459137fa8a3f0c8"} Nov 02 21:13:49.728979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.728936 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" event={"ID":"33a98961-7c4a-4fdd-a2d9-b1072164286d","Type":"ContainerStarted","Data":"51b9571938d7053d08e52481a0b48cf18848ff8c35b912b1c1e90d2a109170d0"} Nov 02 21:13:49.729087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.729072 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:13:49.880748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.880693 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-tnxnd" podStartSLOduration=2.435860412 podStartE2EDuration="6.880678729s" totalImagesPullingTime="4.444818317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:49.879517892 +0000 UTC m=+269.427767672" watchObservedRunningTime="2026-11-02 21:13:49.880678729 +0000 UTC m=+269.428928509" Nov 02 21:13:49.993532 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:49.993431 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" podStartSLOduration=2.5528985669999997 podStartE2EDuration="6.993414921s" totalImagesPullingTime="4.440516354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:49.991854061 +0000 UTC m=+269.540103843" watchObservedRunningTime="2026-11-02 21:13:49.993414921 +0000 UTC m=+269.541664702" Nov 02 21:13:50.043008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.042971 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl"] Nov 02 21:13:50.046482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.046434 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.100070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.100037 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl"] Nov 02 21:13:50.116036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.116006 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pmx7j\"" Nov 02 21:13:50.189854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.189777 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.190146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.189917 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfq6\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-kube-api-access-bdfq6\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.290988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.290885 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfq6\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-kube-api-access-bdfq6\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.290988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.290976 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.351201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.351135 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.352816 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.352761 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfq6\" (UniqueName: \"kubernetes.io/projected/9f23d17e-b8e1-481a-8ce4-e48fc34cb93a-kube-api-access-bdfq6\") pod \"cert-manager-cainjector-5c57bbd4fc-k7wwl\" (UID: \"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.359448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.359411 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" Nov 02 21:13:50.754894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:50.754855 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl"] Nov 02 21:13:50.762249 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:50.762218 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f23d17e_b8e1_481a_8ce4_e48fc34cb93a.slice/crio-b4a810928a053462fa65baa69eb553ab33bddea57fe180a31c369f67af37008b WatchSource:0}: Error finding container b4a810928a053462fa65baa69eb553ab33bddea57fe180a31c369f67af37008b: Status 404 returned error can't find the container with id b4a810928a053462fa65baa69eb553ab33bddea57fe180a31c369f67af37008b Nov 02 21:13:51.674672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.671663 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw"] Nov 02 21:13:51.680687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.679476 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.752152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.752111 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" event={"ID":"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a","Type":"ContainerStarted","Data":"b4a810928a053462fa65baa69eb553ab33bddea57fe180a31c369f67af37008b"} Nov 02 21:13:51.810819 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.810788 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjhn\" (UniqueName: \"kubernetes.io/projected/e7fb50de-9299-49cd-ad74-4557b1f5780a-kube-api-access-7jjhn\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.811314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.810847 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.811314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.810942 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.811314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.811130 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.811314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.811183 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.811314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.811232 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.812404 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.812555 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.813809 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.813869 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.813990 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rhszk\"" Nov 02 21:13:51.814983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.814506 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:13:51.855191 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.855159 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw"] Nov 02 21:13:51.912344 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912306 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.912516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912351 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.912516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912400 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.912516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912462 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjhn\" (UniqueName: \"kubernetes.io/projected/e7fb50de-9299-49cd-ad74-4557b1f5780a-kube-api-access-7jjhn\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.912516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912512 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.912699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.912544 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.914017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.913987 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.916198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.916139 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7fb50de-9299-49cd-ad74-4557b1f5780a-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.916415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.916392 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.916741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.916722 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.918148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.918116 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7fb50de-9299-49cd-ad74-4557b1f5780a-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:51.967852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:51.967747 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjhn\" (UniqueName: \"kubernetes.io/projected/e7fb50de-9299-49cd-ad74-4557b1f5780a-kube-api-access-7jjhn\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-k5klw\" (UID: \"e7fb50de-9299-49cd-ad74-4557b1f5780a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:52.001304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:52.001271 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:52.510536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:52.510448 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw"] Nov 02 21:13:52.521630 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:52.521597 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fb50de_9299_49cd_ad74_4557b1f5780a.slice/crio-65219ce6a47ebde2f9f0313de587d11c6f768232047109a3398d281cb971e76a WatchSource:0}: Error finding container 65219ce6a47ebde2f9f0313de587d11c6f768232047109a3398d281cb971e76a: Status 404 returned error can't find the container with id 65219ce6a47ebde2f9f0313de587d11c6f768232047109a3398d281cb971e76a Nov 02 21:13:52.761155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:52.761073 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" event={"ID":"e7fb50de-9299-49cd-ad74-4557b1f5780a","Type":"ContainerStarted","Data":"65219ce6a47ebde2f9f0313de587d11c6f768232047109a3398d281cb971e76a"} Nov 02 21:13:54.787146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:54.786256 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" event={"ID":"9f23d17e-b8e1-481a-8ce4-e48fc34cb93a","Type":"ContainerStarted","Data":"6dcc96496f1ca6095b6a46dfed94faf5af8a76dfa3feff64cc0f815aad46b71d"} Nov 02 21:13:54.789984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:54.789395 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" event={"ID":"83465063-8244-4e60-8e40-9e2745717cc7","Type":"ContainerStarted","Data":"0e52169704646abb33b695b3c6f9b0e7c7cc7929f3a0fd6d357afd5e363d569c"} Nov 02 21:13:54.790225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:54.790179 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:13:54.901173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:54.901002 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k7wwl" podStartSLOduration=1.885127354 podStartE2EDuration="4.900985166s" totalImagesPullingTime="3.015857812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:54.898713127 +0000 UTC m=+274.446962943" watchObservedRunningTime="2026-11-02 21:13:54.900985166 +0000 UTC m=+274.449234950" Nov 02 21:13:55.007812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:55.007754 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" podStartSLOduration=2.142140742 podStartE2EDuration="7.007739554s" totalImagesPullingTime="4.865598812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:55.006554901 +0000 UTC m=+274.554804706" watchObservedRunningTime="2026-11-02 21:13:55.007739554 +0000 UTC m=+274.555989329" Nov 02 21:13:55.796737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:55.796695 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" event={"ID":"28db1cfd-ba37-4772-91fb-ad9f035e3624","Type":"ContainerStarted","Data":"2100c2f4cb8b85a248112bd8d4755a1ca902d433faa2a8d7efdf19fd5e9e2741"} Nov 02 21:13:55.937302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:55.937225 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qr5f4" podStartSLOduration=2.534102438 podStartE2EDuration="9.937204363s" totalImagesPullingTime="7.403101925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:55.935322691 +0000 UTC m=+275.483572496" watchObservedRunningTime="2026-11-02 21:13:55.937204363 +0000 UTC m=+275.485454146" Nov 02 21:13:55.950368 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:13:55.950295 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:13:56.802485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:56.802404 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" event={"ID":"e7fb50de-9299-49cd-ad74-4557b1f5780a","Type":"ContainerStarted","Data":"a694ee529222c92dbfe27441b302d256795d894725d6d532c438877a6849d3f9"} Nov 02 21:13:57.814947 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:57.814908 5859 generic.go:356] "Generic (PLEG): container finished" podID="e7fb50de-9299-49cd-ad74-4557b1f5780a" containerID="a694ee529222c92dbfe27441b302d256795d894725d6d532c438877a6849d3f9" exitCode=0 Nov 02 21:13:57.815433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:57.815062 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" event={"ID":"e7fb50de-9299-49cd-ad74-4557b1f5780a","Type":"ContainerDied","Data":"a694ee529222c92dbfe27441b302d256795d894725d6d532c438877a6849d3f9"} Nov 02 21:13:58.351248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.351216 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-6pljp"] Nov 02 21:13:58.354991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.354969 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.410203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.410172 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9n5mr\"" Nov 02 21:13:58.411849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.411823 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6pljp"] Nov 02 21:13:58.480823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.480797 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcr92\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-kube-api-access-xcr92\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.480972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.480856 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-bound-sa-token\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.582275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.582241 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-bound-sa-token\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.582447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.582314 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcr92\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-kube-api-access-xcr92\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.636390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.636321 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-bound-sa-token\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.636600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.636577 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcr92\" (UniqueName: \"kubernetes.io/projected/b21c8e35-e416-45f7-b7ab-e5bfbf524990-kube-api-access-xcr92\") pod \"cert-manager-597b95cb84-6pljp\" (UID: \"b21c8e35-e416-45f7-b7ab-e5bfbf524990\") " pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.666370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.666343 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6pljp" Nov 02 21:13:58.821855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.821825 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" event={"ID":"e7fb50de-9299-49cd-ad74-4557b1f5780a","Type":"ContainerStarted","Data":"c4052ba5144facf168987996adc5547a3c510c55e6bccc37675593082d7f42b5"} Nov 02 21:13:58.822179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.821975 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:13:58.943575 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:58.943486 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" podStartSLOduration=2.717549269 podStartE2EDuration="7.943472582s" totalImagesPullingTime="4.759368499s" totalInitContainerRuntime="466.554814ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:58.94193366 +0000 UTC m=+278.490183461" watchObservedRunningTime="2026-11-02 21:13:58.943472582 +0000 UTC m=+278.491722362" Nov 02 21:13:59.106870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:59.106844 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6pljp"] Nov 02 21:13:59.113717 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:13:59.113693 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb21c8e35_e416_45f7_b7ab_e5bfbf524990.slice/crio-78975a98322a3413873a854688408d6a1951eab4e82f808e8117be8075c6fb36 WatchSource:0}: Error finding container 78975a98322a3413873a854688408d6a1951eab4e82f808e8117be8075c6fb36: Status 404 returned error can't find the container with id 78975a98322a3413873a854688408d6a1951eab4e82f808e8117be8075c6fb36 Nov 02 21:13:59.827666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:59.827626 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6pljp" event={"ID":"b21c8e35-e416-45f7-b7ab-e5bfbf524990","Type":"ContainerStarted","Data":"4e6876d34a9df841a5a1ffb872e82101820213059019c02b38e2a13a85e08181"} Nov 02 21:13:59.828097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:59.827678 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6pljp" event={"ID":"b21c8e35-e416-45f7-b7ab-e5bfbf524990","Type":"ContainerStarted","Data":"78975a98322a3413873a854688408d6a1951eab4e82f808e8117be8075c6fb36"} Nov 02 21:13:59.940517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:13:59.940468 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-6pljp" podStartSLOduration=1.9404497630000002 podStartE2EDuration="1.940449763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:59.939409991 +0000 UTC m=+279.487659790" watchObservedRunningTime="2026-11-02 21:13:59.940449763 +0000 UTC m=+279.488699545" Nov 02 21:14:00.740012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:00.739983 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-vsgm4" Nov 02 21:14:01.804711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:01.804636 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-flz5d" Nov 02 21:14:09.829897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:09.829868 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-k5klw" Nov 02 21:14:23.210808 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.210769 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb"] Nov 02 21:14:23.222284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.222260 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.271390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.271362 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:14:23.291509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.291482 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e594ed53-c428-4eb4-9410-cca9c7a23763-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.291605 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.291521 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zzc\" (UniqueName: \"kubernetes.io/projected/e594ed53-c428-4eb4-9410-cca9c7a23763-kube-api-access-j9zzc\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.291702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.291680 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e594ed53-c428-4eb4-9410-cca9c7a23763-tmp\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.293773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.293754 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:14:23.319059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.319033 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb"] Nov 02 21:14:23.321409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.321344 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 21:14:23.321539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.321451 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-62vv5\"" Nov 02 21:14:23.321539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.321518 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 21:14:23.392920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.392868 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zzc\" (UniqueName: \"kubernetes.io/projected/e594ed53-c428-4eb4-9410-cca9c7a23763-kube-api-access-j9zzc\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.393157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.392999 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e594ed53-c428-4eb4-9410-cca9c7a23763-tmp\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.393157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.393030 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e594ed53-c428-4eb4-9410-cca9c7a23763-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.395241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.395219 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e594ed53-c428-4eb4-9410-cca9c7a23763-tmp\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.395499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.395479 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e594ed53-c428-4eb4-9410-cca9c7a23763-tls-certs\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.446353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.446321 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zzc\" (UniqueName: \"kubernetes.io/projected/e594ed53-c428-4eb4-9410-cca9c7a23763-kube-api-access-j9zzc\") pod \"kube-auth-proxy-78fbf7b575-mt2tb\" (UID: \"e594ed53-c428-4eb4-9410-cca9c7a23763\") " pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.533684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.533600 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" Nov 02 21:14:23.901999 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.901963 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb"] Nov 02 21:14:23.907645 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:14:23.907621 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode594ed53_c428_4eb4_9410_cca9c7a23763.slice/crio-e6e5d98c9eac93bc66f5591018523668889b314fd1f5d8f9c349a0987754b1bc WatchSource:0}: Error finding container e6e5d98c9eac93bc66f5591018523668889b314fd1f5d8f9c349a0987754b1bc: Status 404 returned error can't find the container with id e6e5d98c9eac93bc66f5591018523668889b314fd1f5d8f9c349a0987754b1bc Nov 02 21:14:23.942641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:23.942610 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" event={"ID":"e594ed53-c428-4eb4-9410-cca9c7a23763","Type":"ContainerStarted","Data":"e6e5d98c9eac93bc66f5591018523668889b314fd1f5d8f9c349a0987754b1bc"} Nov 02 21:14:25.947232 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:14:25.947196 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:14:26.472238 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.472191 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff"] Nov 02 21:14:26.484434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.484406 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.522841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.522801 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-cert\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.523018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.522856 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-metrics-certs\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.523018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.522973 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd80e926-63ac-4804-9364-b7458f760aa8-manager-config\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.523137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.523066 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7vq\" (UniqueName: \"kubernetes.io/projected/bd80e926-63ac-4804-9364-b7458f760aa8-kube-api-access-sh7vq\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.530390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.530363 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff"] Nov 02 21:14:26.574964 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.574927 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 21:14:26.580286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.579991 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 21:14:26.580286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.579994 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-zq4hf\"" Nov 02 21:14:26.580286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.580195 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 21:14:26.624250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.624218 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-cert\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.624416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.624344 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-metrics-certs\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.624416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.624394 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd80e926-63ac-4804-9364-b7458f760aa8-manager-config\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.624851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.624826 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7vq\" (UniqueName: \"kubernetes.io/projected/bd80e926-63ac-4804-9364-b7458f760aa8-kube-api-access-sh7vq\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.625367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.625342 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd80e926-63ac-4804-9364-b7458f760aa8-manager-config\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.626986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.626938 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-metrics-certs\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.627085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.627041 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd80e926-63ac-4804-9364-b7458f760aa8-cert\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.678337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.678303 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7vq\" (UniqueName: \"kubernetes.io/projected/bd80e926-63ac-4804-9364-b7458f760aa8-kube-api-access-sh7vq\") pod \"jobset-controller-manager-64cfd85455-8q5ff\" (UID: \"bd80e926-63ac-4804-9364-b7458f760aa8\") " pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:26.802334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:26.802248 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:27.172003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:27.171945 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff"] Nov 02 21:14:27.190091 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:14:27.190064 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd80e926_63ac_4804_9364_b7458f760aa8.slice/crio-88c7824a152e29ed28b3344921516d862619ab6874862994f3b3252215092515 WatchSource:0}: Error finding container 88c7824a152e29ed28b3344921516d862619ab6874862994f3b3252215092515: Status 404 returned error can't find the container with id 88c7824a152e29ed28b3344921516d862619ab6874862994f3b3252215092515 Nov 02 21:14:27.963575 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:27.963532 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" event={"ID":"e594ed53-c428-4eb4-9410-cca9c7a23763","Type":"ContainerStarted","Data":"ea727776254a5bc256f62adc0a6a257ff6621328bf299f45a8268aba0bb5b9e1"} Nov 02 21:14:27.964787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:27.964760 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" event={"ID":"bd80e926-63ac-4804-9364-b7458f760aa8","Type":"ContainerStarted","Data":"88c7824a152e29ed28b3344921516d862619ab6874862994f3b3252215092515"} Nov 02 21:14:28.070970 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:28.070898 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78fbf7b575-mt2tb" podStartSLOduration=1.476134829 podStartE2EDuration="5.070884177s" totalImagesPullingTime="3.594749348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:28.068878897 +0000 UTC m=+307.617128681" watchObservedRunningTime="2026-11-02 21:14:28.070884177 +0000 UTC m=+307.619133958" Nov 02 21:14:31.984934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:31.984904 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" event={"ID":"bd80e926-63ac-4804-9364-b7458f760aa8","Type":"ContainerStarted","Data":"75dbb837b5f37fb4c5fa2c0fde81ebb92c805f52bfcab72eab34e5fdc2bc6c34"} Nov 02 21:14:31.985331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:31.985023 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:32.093223 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:32.093177 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" podStartSLOduration=1.413022113 podStartE2EDuration="6.093163095s" totalImagesPullingTime="4.680140982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:32.091390981 +0000 UTC m=+311.639640773" watchObservedRunningTime="2026-11-02 21:14:32.093163095 +0000 UTC m=+311.641412877" Nov 02 21:14:42.995439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:14:42.995412 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64cfd85455-8q5ff" Nov 02 21:14:55.944250 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:14:55.944205 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:15:25.944197 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:15:25.944151 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:15:43.239296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.239223 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mqd69"] Nov 02 21:15:43.242731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.242709 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.291681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.291658 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:15:43.311373 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.311340 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcn7z\" (UniqueName: \"kubernetes.io/projected/5a272ed8-590a-4bc5-aace-1f092f17eb9d-kube-api-access-gcn7z\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.311485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.311400 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-cert\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.311537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.311494 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-metrics-certs\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.338330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.338311 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-86vbb\"" Nov 02 21:15:43.338439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.338326 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 21:15:43.338439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.338344 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:15:43.338439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.338374 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 21:15:43.340055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.340034 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mqd69"] Nov 02 21:15:43.412382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.412358 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-metrics-certs\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.412478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.412404 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcn7z\" (UniqueName: \"kubernetes.io/projected/5a272ed8-590a-4bc5-aace-1f092f17eb9d-kube-api-access-gcn7z\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.412478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.412444 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-cert\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.414679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.414661 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-cert\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.414770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.414704 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a272ed8-590a-4bc5-aace-1f092f17eb9d-metrics-certs\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.467694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.467677 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcn7z\" (UniqueName: \"kubernetes.io/projected/5a272ed8-590a-4bc5-aace-1f092f17eb9d-kube-api-access-gcn7z\") pod \"odh-model-controller-769875db56-mqd69\" (UID: \"5a272ed8-590a-4bc5-aace-1f092f17eb9d\") " pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.555004 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.554961 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:43.942840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.942803 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mqd69"] Nov 02 21:15:43.949497 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:15:43.949473 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a272ed8_590a_4bc5_aace_1f092f17eb9d.slice/crio-81d80b09b607cece654a79347fe6b01ff24115f994ecd49238a21aadb6324b19 WatchSource:0}: Error finding container 81d80b09b607cece654a79347fe6b01ff24115f994ecd49238a21aadb6324b19: Status 404 returned error can't find the container with id 81d80b09b607cece654a79347fe6b01ff24115f994ecd49238a21aadb6324b19 Nov 02 21:15:43.950739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:43.950721 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:15:44.316988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:44.316942 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mqd69" event={"ID":"5a272ed8-590a-4bc5-aace-1f092f17eb9d","Type":"ContainerStarted","Data":"81d80b09b607cece654a79347fe6b01ff24115f994ecd49238a21aadb6324b19"} Nov 02 21:15:55.370436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:55.370401 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mqd69" event={"ID":"5a272ed8-590a-4bc5-aace-1f092f17eb9d","Type":"ContainerStarted","Data":"66c0c910737445afc802884b90675ed72e76cd53fc933ee8a093f81b836814a4"} Nov 02 21:15:55.370863 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:55.370514 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:15:55.478573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:55.478488 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mqd69" podStartSLOduration=1.294284487 podStartE2EDuration="12.478475958s" totalImagesPullingTime="11.184191471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:55.476818814 +0000 UTC m=+395.025068612" watchObservedRunningTime="2026-11-02 21:15:55.478475958 +0000 UTC m=+395.026725739" Nov 02 21:15:55.944606 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:15:55.944564 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:15:58.728792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.728756 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:15:58.732312 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.732293 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.781661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.781638 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:15:58.828290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.828256 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:15:58.828439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.828296 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:15:58.828439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.828296 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 21:15:58.828439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.828376 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 21:15:58.829913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.829896 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:15:58.849897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.849874 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.850033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.849901 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.850033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.850003 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.850140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.850119 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpv6\" (UniqueName: \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.951013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.950983 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.951206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.951038 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpv6\" (UniqueName: \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.951206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.951077 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.951206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.951092 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.951740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.951717 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.953336 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.953316 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:58.953428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:58.953351 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:59.004983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:59.004902 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpv6\" (UniqueName: \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\") pod \"maas-api-bfd69c7d4-t8rtr\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:59.045172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:59.045138 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:15:59.413023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:15:59.412988 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:15:59.419108 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:15:59.419081 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657110ae_6a03_46d5_a13e_4007dc811bf6.slice/crio-3b49359dcb4feb35b124a642aed4fae7d71dc572c31581f4430a263682646657 WatchSource:0}: Error finding container 3b49359dcb4feb35b124a642aed4fae7d71dc572c31581f4430a263682646657: Status 404 returned error can't find the container with id 3b49359dcb4feb35b124a642aed4fae7d71dc572c31581f4430a263682646657 Nov 02 21:16:00.394353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:00.394315 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" event={"ID":"657110ae-6a03-46d5-a13e-4007dc811bf6","Type":"ContainerStarted","Data":"3b49359dcb4feb35b124a642aed4fae7d71dc572c31581f4430a263682646657"} Nov 02 21:16:06.377290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:06.377262 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mqd69" Nov 02 21:16:06.424857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:06.424823 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" event={"ID":"657110ae-6a03-46d5-a13e-4007dc811bf6","Type":"ContainerStarted","Data":"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198"} Nov 02 21:16:06.425015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:06.424879 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:16:06.595095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:06.595043 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" podStartSLOduration=2.091346512 podStartE2EDuration="8.595027105s" totalImagesPullingTime="6.503680593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:06.593298605 +0000 UTC m=+406.141548386" watchObservedRunningTime="2026-11-02 21:16:06.595027105 +0000 UTC m=+406.143276886" Nov 02 21:16:12.434666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:12.434640 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:16:13.781895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.781862 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:16:13.785589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.785570 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.834722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.834701 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:16:13.834722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.834709 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\"" Nov 02 21:16:13.882632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.882601 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:16:13.882891 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.882871 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.882933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.882901 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.882994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.882973 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883012 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883040 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883138 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883116 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883148 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6vp\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883170 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.883289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.883272 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.983927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.983902 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.983931 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.983967 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984008 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984115 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6vp\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984163 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984205 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984284 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984308 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984339 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984390 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984469 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.984690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.984622 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.985044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.985019 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.986249 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.986226 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:13.986846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:13.986827 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:14.037072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:14.037020 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:14.037821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:14.037803 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6vp\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ckwj9\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:14.098545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:14.098526 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:14.512660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:14.512630 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:16:14.557141 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:16:14.557113 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0d477e_a2b7_42d6_ba58_84053a0ac5ec.slice/crio-19c6586c1d91514097f0eeb23d2df75f6af4b3057a850c5ab7348aef1c042655 WatchSource:0}: Error finding container 19c6586c1d91514097f0eeb23d2df75f6af4b3057a850c5ab7348aef1c042655: Status 404 returned error can't find the container with id 19c6586c1d91514097f0eeb23d2df75f6af4b3057a850c5ab7348aef1c042655 Nov 02 21:16:15.466261 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:15.466224 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" event={"ID":"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec","Type":"ContainerStarted","Data":"19c6586c1d91514097f0eeb23d2df75f6af4b3057a850c5ab7348aef1c042655"} Nov 02 21:16:22.366536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:22.366503 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:22.366847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:22.366571 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:22.366847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:22.366599 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:22.503429 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:22.503392 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" event={"ID":"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec","Type":"ContainerStarted","Data":"a6cd05ede80cc239c39eb4718845d4b0e8fe401b32d6954c62ec4970be0a348f"} Nov 02 21:16:22.628752 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:22.628646 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podStartSLOduration=1.822108563 podStartE2EDuration="9.628629457s" totalImagesPullingTime="7.806520894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:22.626361306 +0000 UTC m=+422.174611108" watchObservedRunningTime="2026-11-02 21:16:22.628629457 +0000 UTC m=+422.176879240" Nov 02 21:16:23.098934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.098904 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:23.100342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.100316 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:23.100405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.100379 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:23.429800 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.429705 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:16:23.430300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.430259 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" podUID="657110ae-6a03-46d5-a13e-4007dc811bf6" containerName="maas-api" containerID="cri-o://0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198" gracePeriod=30 Nov 02 21:16:23.685807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.685754 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:16:23.775306 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775279 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\") pod \"657110ae-6a03-46d5-a13e-4007dc811bf6\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " Nov 02 21:16:23.775470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775351 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\") pod \"657110ae-6a03-46d5-a13e-4007dc811bf6\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " Nov 02 21:16:23.775539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775506 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\" (UniqueName: \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\") pod \"657110ae-6a03-46d5-a13e-4007dc811bf6\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " Nov 02 21:16:23.775593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775554 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\") pod \"657110ae-6a03-46d5-a13e-4007dc811bf6\" (UID: \"657110ae-6a03-46d5-a13e-4007dc811bf6\") " Nov 02 21:16:23.775667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775649 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca" pod "657110ae-6a03-46d5-a13e-4007dc811bf6" (UID: "657110ae-6a03-46d5-a13e-4007dc811bf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:23.775853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.775832 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/657110ae-6a03-46d5-a13e-4007dc811bf6-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:23.777521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.777486 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6" pod "657110ae-6a03-46d5-a13e-4007dc811bf6" (UID: "657110ae-6a03-46d5-a13e-4007dc811bf6"). InnerVolumeSpecName "kube-api-access-hvpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:23.777609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.777487 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls" pod "657110ae-6a03-46d5-a13e-4007dc811bf6" (UID: "657110ae-6a03-46d5-a13e-4007dc811bf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:23.777609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.777519 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs" pod "657110ae-6a03-46d5-a13e-4007dc811bf6" (UID: "657110ae-6a03-46d5-a13e-4007dc811bf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:23.876682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.876653 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpv6\" (UniqueName: \"kubernetes.io/projected/657110ae-6a03-46d5-a13e-4007dc811bf6-kube-api-access-hvpv6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:23.876682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.876678 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:23.876682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:23.876688 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/657110ae-6a03-46d5-a13e-4007dc811bf6-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:24.098778 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.098747 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:24.098920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.098793 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:24.543260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.543182 5859 generic.go:356] "Generic (PLEG): container finished" podID="657110ae-6a03-46d5-a13e-4007dc811bf6" containerID="0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198" exitCode=0 Nov 02 21:16:24.543260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.543241 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" Nov 02 21:16:24.543750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.543264 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" event={"ID":"657110ae-6a03-46d5-a13e-4007dc811bf6","Type":"ContainerDied","Data":"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198"} Nov 02 21:16:24.543750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.543316 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr" event={"ID":"657110ae-6a03-46d5-a13e-4007dc811bf6","Type":"ContainerDied","Data":"3b49359dcb4feb35b124a642aed4fae7d71dc572c31581f4430a263682646657"} Nov 02 21:16:24.543750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.543339 5859 scope.go:122] "RemoveContainer" containerID="0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198" Nov 02 21:16:24.552184 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.552166 5859 scope.go:122] "RemoveContainer" containerID="0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198" Nov 02 21:16:24.552443 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:16:24.552424 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198\": container with ID starting with 0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198 not found: ID does not exist" containerID="0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198" Nov 02 21:16:24.552487 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.552452 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198"} err="failed to get container status \"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198\": rpc error: code = NotFound desc = could not find container \"0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198\": container with ID starting with 0ecca04f559b93b5e25d08b0a68d03ff1b4ca078fdfee34805862c3566900198 not found: ID does not exist" Nov 02 21:16:24.695776 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.695748 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:16:24.741789 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:24.741757 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bfd69c7d4-t8rtr"] Nov 02 21:16:25.099721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:25.099691 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:25.099890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:25.099741 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:25.952772 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:16:25.952730 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:16:26.099037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:26.099005 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:26.099193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:26.099054 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:26.568279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:26.568244 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657110ae-6a03-46d5-a13e-4007dc811bf6" path="/var/lib/kubelet/pods/657110ae-6a03-46d5-a13e-4007dc811bf6/volumes" Nov 02 21:16:27.099032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:27.099004 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:27.099407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:27.099055 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:28.099807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:28.099769 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ckwj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 21:16:28.100321 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:28.099837 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 21:16:29.099021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:29.098988 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:29.103043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:29.103014 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:29.567723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:16:29.567700 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:16:55.942776 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:16:55.942734 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:17:25.942654 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:17:25.942617 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:17:55.943624 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:17:55.943582 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:18:13.345961 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.345927 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm"] Nov 02 21:18:13.348447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.346368 5859 cpu_manager.go:414] "containerMap: removing container" podUID="657110ae-6a03-46d5-a13e-4007dc811bf6" containerName="maas-api" Nov 02 21:18:13.348447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.346384 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657110ae-6a03-46d5-a13e-4007dc811bf6" containerName="maas-api" Nov 02 21:18:13.348447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.346463 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="657110ae-6a03-46d5-a13e-4007dc811bf6" containerName="maas-api" Nov 02 21:18:13.352056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.350789 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.402550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.402496 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 21:18:13.421068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.421041 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 21:18:13.448564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.448527 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4dfx7\"" Nov 02 21:18:13.448737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.448621 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 21:18:13.450149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.450128 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm"] Nov 02 21:18:13.482618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482579 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.482618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482616 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q5jq\" (UniqueName: \"kubernetes.io/projected/1113b3ed-88cb-4927-9acd-182acfc8c366-kube-api-access-9q5jq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.482820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482731 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.482820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482767 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113b3ed-88cb-4927-9acd-182acfc8c366-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.482882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482863 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.482917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.482884 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584389 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584344 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113b3ed-88cb-4927-9acd-182acfc8c366-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584410 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584432 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584492 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584516 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5jq\" (UniqueName: \"kubernetes.io/projected/1113b3ed-88cb-4927-9acd-182acfc8c366-kube-api-access-9q5jq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584585 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.584918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.585032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.584933 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.585080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.585030 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.587296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.587274 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1113b3ed-88cb-4927-9acd-182acfc8c366-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.587487 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.587468 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1113b3ed-88cb-4927-9acd-182acfc8c366-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.638030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.637940 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5jq\" (UniqueName: \"kubernetes.io/projected/1113b3ed-88cb-4927-9acd-182acfc8c366-kube-api-access-9q5jq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm\" (UID: \"1113b3ed-88cb-4927-9acd-182acfc8c366\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:13.665643 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:13.665613 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:14.031278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:14.031198 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm"] Nov 02 21:18:14.037097 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:18:14.037065 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1113b3ed_88cb_4927_9acd_182acfc8c366.slice/crio-8ceb534c3f00bc7ca100fd8ebc940c37183646bcb57ae7e7fe223f0154920de0 WatchSource:0}: Error finding container 8ceb534c3f00bc7ca100fd8ebc940c37183646bcb57ae7e7fe223f0154920de0: Status 404 returned error can't find the container with id 8ceb534c3f00bc7ca100fd8ebc940c37183646bcb57ae7e7fe223f0154920de0 Nov 02 21:18:15.046741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:15.046694 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" event={"ID":"1113b3ed-88cb-4927-9acd-182acfc8c366","Type":"ContainerStarted","Data":"8ceb534c3f00bc7ca100fd8ebc940c37183646bcb57ae7e7fe223f0154920de0"} Nov 02 21:18:16.054026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:16.053992 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" event={"ID":"1113b3ed-88cb-4927-9acd-182acfc8c366","Type":"ContainerStarted","Data":"878901ca052c81ff3205929b584cc54f5fecf6576a26d364d2c467a1301022ab"} Nov 02 21:18:16.163345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:16.163289 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" podStartSLOduration=1.233025159 podStartE2EDuration="3.163275168s" totalImagesPullingTime="1.930250009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:16.162197244 +0000 UTC m=+535.710447037" watchObservedRunningTime="2026-11-02 21:18:16.163275168 +0000 UTC m=+535.711524949" Nov 02 21:18:16.666003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:16.665941 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:19.261046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.261011 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s"] Nov 02 21:18:19.264683 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.264665 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.318782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.318755 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s"] Nov 02 21:18:19.341962 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.341931 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrg6\" (UniqueName: \"kubernetes.io/projected/a6280fe2-f94e-449b-9eed-18159ae4611f-kube-api-access-gmrg6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.342086 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.342011 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6280fe2-f94e-449b-9eed-18159ae4611f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.342086 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.342061 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.342158 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.342139 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.342218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.342206 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.342269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.342258 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.359941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.359914 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 21:18:19.443212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443182 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrg6\" (UniqueName: \"kubernetes.io/projected/a6280fe2-f94e-449b-9eed-18159ae4611f-kube-api-access-gmrg6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443231 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6280fe2-f94e-449b-9eed-18159ae4611f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443282 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443330 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443372 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443424 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443690 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443716 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.443803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.443768 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.445661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.445631 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6280fe2-f94e-449b-9eed-18159ae4611f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.445815 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.445797 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6280fe2-f94e-449b-9eed-18159ae4611f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.496316 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.496293 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrg6\" (UniqueName: \"kubernetes.io/projected/a6280fe2-f94e-449b-9eed-18159ae4611f-kube-api-access-gmrg6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bvq5s\" (UID: \"a6280fe2-f94e-449b-9eed-18159ae4611f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.577412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.577388 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:19.942158 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:19.942079 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s"] Nov 02 21:18:19.947843 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:18:19.947814 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6280fe2_f94e_449b_9eed_18159ae4611f.slice/crio-9efa26beda5a9ce378b89255587d1396d4f30539dd614858380199e97d9ac5ae WatchSource:0}: Error finding container 9efa26beda5a9ce378b89255587d1396d4f30539dd614858380199e97d9ac5ae: Status 404 returned error can't find the container with id 9efa26beda5a9ce378b89255587d1396d4f30539dd614858380199e97d9ac5ae Nov 02 21:18:20.075620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:20.075582 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" event={"ID":"a6280fe2-f94e-449b-9eed-18159ae4611f","Type":"ContainerStarted","Data":"9efa26beda5a9ce378b89255587d1396d4f30539dd614858380199e97d9ac5ae"} Nov 02 21:18:21.081229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:21.081197 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" event={"ID":"a6280fe2-f94e-449b-9eed-18159ae4611f","Type":"ContainerStarted","Data":"f9d38bf97aedad47b234599d302d0ac55f34a94a433a3aceac8f4f664d3ada22"} Nov 02 21:18:21.190446 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:21.190403 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" podStartSLOduration=1.5665689139999999 podStartE2EDuration="2.190390741s" totalImagesPullingTime="623.821827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:21.188599804 +0000 UTC m=+540.736849595" watchObservedRunningTime="2026-11-02 21:18:21.190390741 +0000 UTC m=+540.738640522" Nov 02 21:18:21.578469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:21.578440 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:23.666724 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:23.666692 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:23.679729 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:23.679702 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:24.105714 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:24.105687 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm" Nov 02 21:18:25.943250 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:18:25.943216 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:18:29.577977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:29.577908 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:29.589907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:29.589879 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:30.138774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:30.138743 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bvq5s" Nov 02 21:18:31.005830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.005798 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x"] Nov 02 21:18:31.012233 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.012212 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053465 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053584 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053620 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053689 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480e96b6-2aae-45f4-b3d3-98840fb956f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053736 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.053806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.053762 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/480e96b6-2aae-45f4-b3d3-98840fb956f6-kube-api-access-57wpr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.109987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.109941 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 21:18:31.117510 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.113724 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x"] Nov 02 21:18:31.154298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154261 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154475 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154372 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154475 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154406 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154520 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480e96b6-2aae-45f4-b3d3-98840fb956f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154552 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154592 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/480e96b6-2aae-45f4-b3d3-98840fb956f6-kube-api-access-57wpr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154702 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.154897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.154875 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.155028 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.155000 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.156736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.156712 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480e96b6-2aae-45f4-b3d3-98840fb956f6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.157320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.157293 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480e96b6-2aae-45f4-b3d3-98840fb956f6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.207729 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.207698 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wpr\" (UniqueName: \"kubernetes.io/projected/480e96b6-2aae-45f4-b3d3-98840fb956f6-kube-api-access-57wpr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x\" (UID: \"480e96b6-2aae-45f4-b3d3-98840fb956f6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.326022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.325996 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:31.694766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:31.694685 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x"] Nov 02 21:18:31.700142 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:18:31.700094 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480e96b6_2aae_45f4_b3d3_98840fb956f6.slice/crio-7dcda7fc7d4db3d9008ec7a1717928f8db49801402fc6fd167eb1063a6a7d9ee WatchSource:0}: Error finding container 7dcda7fc7d4db3d9008ec7a1717928f8db49801402fc6fd167eb1063a6a7d9ee: Status 404 returned error can't find the container with id 7dcda7fc7d4db3d9008ec7a1717928f8db49801402fc6fd167eb1063a6a7d9ee Nov 02 21:18:32.141644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:32.141606 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" event={"ID":"480e96b6-2aae-45f4-b3d3-98840fb956f6","Type":"ContainerStarted","Data":"7dcda7fc7d4db3d9008ec7a1717928f8db49801402fc6fd167eb1063a6a7d9ee"} Nov 02 21:18:33.147906 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:33.147868 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" event={"ID":"480e96b6-2aae-45f4-b3d3-98840fb956f6","Type":"ContainerStarted","Data":"ef694eda284bcae1b5b5c17a714cb69793a12755b2d34f6ca4e1334a4c86060a"} Nov 02 21:18:33.257571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:33.257514 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" podStartSLOduration=2.637207472 podStartE2EDuration="3.25749613s" totalImagesPullingTime="620.288658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:33.255696452 +0000 UTC m=+552.803946252" watchObservedRunningTime="2026-11-02 21:18:33.25749613 +0000 UTC m=+552.805745910" Nov 02 21:18:33.326545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:33.326497 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:41.327268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:41.327234 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:41.339206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:41.339180 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:42.204544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:42.204514 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x" Nov 02 21:18:44.040870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.040838 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n"] Nov 02 21:18:44.044513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.044493 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.139548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.139518 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 21:18:44.141366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.141341 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n"] Nov 02 21:18:44.172925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.172899 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0ac144-950e-424b-9461-6a2333d05969-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.173077 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.172935 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.173077 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.173062 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws5h\" (UniqueName: \"kubernetes.io/projected/8e0ac144-950e-424b-9461-6a2333d05969-kube-api-access-zws5h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.173178 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.173104 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.173213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.173186 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.173251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.173209 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274103 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274074 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274124 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274152 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274250 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0ac144-950e-424b-9461-6a2333d05969-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274289 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274344 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zws5h\" (UniqueName: \"kubernetes.io/projected/8e0ac144-950e-424b-9461-6a2333d05969-kube-api-access-zws5h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274508 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274565 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.274683 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.274663 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.276300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.276280 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0ac144-950e-424b-9461-6a2333d05969-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.276548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.276531 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0ac144-950e-424b-9461-6a2333d05969-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.327268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.327242 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws5h\" (UniqueName: \"kubernetes.io/projected/8e0ac144-950e-424b-9461-6a2333d05969-kube-api-access-zws5h\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n\" (UID: \"8e0ac144-950e-424b-9461-6a2333d05969\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.355744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.355720 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:44.724827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.724757 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n"] Nov 02 21:18:44.729963 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:18:44.729923 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e0ac144_950e_424b_9461_6a2333d05969.slice/crio-a54d1a02ea974bb9a30ea7d8db8e24603f66209f6605a23391c8105dc8f18860 WatchSource:0}: Error finding container a54d1a02ea974bb9a30ea7d8db8e24603f66209f6605a23391c8105dc8f18860: Status 404 returned error can't find the container with id a54d1a02ea974bb9a30ea7d8db8e24603f66209f6605a23391c8105dc8f18860 Nov 02 21:18:44.877094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.877066 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:18:44.881823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.881802 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.963197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.963148 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:18:44.981196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981118 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981163 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981270 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981300 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981412 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981479 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981530 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981635 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:44.981717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:44.981682 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083311 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083360 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083396 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083417 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083476 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083505 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083544 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083571 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083631 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.083764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083763 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.084315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.083837 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.084315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.084190 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.084315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.084228 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.084445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.084331 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.085754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.085734 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.085885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.085869 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.136785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.136758 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.136929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.136896 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gkvm4\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.193972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.193920 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:45.210790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.210761 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" event={"ID":"8e0ac144-950e-424b-9461-6a2333d05969","Type":"ContainerStarted","Data":"a54d1a02ea974bb9a30ea7d8db8e24603f66209f6605a23391c8105dc8f18860"} Nov 02 21:18:45.662751 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.662718 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:18:45.668517 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:18:45.668485 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1386322d_1fb7_4444_988f_4e4942c2d262.slice/crio-17ddcb69d9cf7c86e2fb6833db6c7e96bc0ad39cbd44904682078d7fa2b10340 WatchSource:0}: Error finding container 17ddcb69d9cf7c86e2fb6833db6c7e96bc0ad39cbd44904682078d7fa2b10340: Status 404 returned error can't find the container with id 17ddcb69d9cf7c86e2fb6833db6c7e96bc0ad39cbd44904682078d7fa2b10340 Nov 02 21:18:45.670967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.670914 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:18:45.671054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.671035 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:18:45.671126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:45.671111 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:18:46.233323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.233283 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" event={"ID":"8e0ac144-950e-424b-9461-6a2333d05969","Type":"ContainerStarted","Data":"01bf96b0c845bc383889ec32871033e3a86e967104fedaf667a3ccb1cb0c2f95"} Nov 02 21:18:46.235148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.235119 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" event={"ID":"1386322d-1fb7-4444-988f-4e4942c2d262","Type":"ContainerStarted","Data":"2097bbf2bcca441f8483d66004360056710f375c7ec2ed46e32935bd8affc747"} Nov 02 21:18:46.235273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.235173 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" event={"ID":"1386322d-1fb7-4444-988f-4e4942c2d262","Type":"ContainerStarted","Data":"17ddcb69d9cf7c86e2fb6833db6c7e96bc0ad39cbd44904682078d7fa2b10340"} Nov 02 21:18:46.340808 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.340750 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" podStartSLOduration=1.714102763 podStartE2EDuration="2.340734758s" totalImagesPullingTime="626.631995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:46.339732923 +0000 UTC m=+565.887982702" watchObservedRunningTime="2026-11-02 21:18:46.340734758 +0000 UTC m=+565.888984538" Nov 02 21:18:46.356515 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.356488 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:46.449934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:46.449871 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podStartSLOduration=2.449856404 podStartE2EDuration="2.449856404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:46.448656999 +0000 UTC m=+565.996906817" watchObservedRunningTime="2026-11-02 21:18:46.449856404 +0000 UTC m=+565.998106291" Nov 02 21:18:47.194896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:47.194859 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:47.196113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:47.196086 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:47.196228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:47.196144 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:48.194770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:48.194737 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:48.195196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:48.194799 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:49.194824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:49.194784 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:49.195257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:49.194861 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:50.194857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:50.194827 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:50.195308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:50.194870 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:51.194681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:51.194644 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:51.194871 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:51.194696 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:52.194677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:52.194642 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 21:18:52.195114 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:52.194705 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 21:18:54.236147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:54.236099 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gkvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:18:54.236642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:54.236187 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:18:54.356542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:54.356492 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:54.374654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:54.374620 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:55.044553 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:55.044526 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:55.044827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:55.044809 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:55.045410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:55.045391 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:18:55.291493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:55.291471 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:18:55.943750 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:18:55.943713 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:18:59.988451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:59.988420 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:18:59.991905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:18:59.991886 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152477 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152543 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152616 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152670 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152756 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152780 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.152840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152813 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.153006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152875 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.153006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.152911 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4mk\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.184410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.184381 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:19:00.253655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253629 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253663 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253693 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253734 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253751 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253776 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.253817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253802 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253834 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4mk\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.253939 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.254253 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254444 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.254423 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.254444 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254560 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.254515 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.254724 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.254709 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.255880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.255861 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.256213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.256192 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.307971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.307924 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.308058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.307935 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4mk\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-bd2cj\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:00.604931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:00.604901 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:01.023711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.023629 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:19:01.029274 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:01.029247 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12df31b9_59d7_4528_869a_7f179fe61e65.slice/crio-69eff0dfb1d1e261208d8f03cd9e53f1a3461570b99f006b40c1637c886b015d WatchSource:0}: Error finding container 69eff0dfb1d1e261208d8f03cd9e53f1a3461570b99f006b40c1637c886b015d: Status 404 returned error can't find the container with id 69eff0dfb1d1e261208d8f03cd9e53f1a3461570b99f006b40c1637c886b015d Nov 02 21:19:01.031433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.031402 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:01.031489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.031472 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:01.031519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.031501 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:01.313100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.313013 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" event={"ID":"12df31b9-59d7-4528-869a-7f179fe61e65","Type":"ContainerStarted","Data":"7273f8710f44ff549b91654b616779f7923a69fce3a6574bf541d95c5c33a77d"} Nov 02 21:19:01.313100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.313066 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" event={"ID":"12df31b9-59d7-4528-869a-7f179fe61e65","Type":"ContainerStarted","Data":"69eff0dfb1d1e261208d8f03cd9e53f1a3461570b99f006b40c1637c886b015d"} Nov 02 21:19:01.423258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.423201 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podStartSLOduration=2.423186464 podStartE2EDuration="2.423186464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:01.420876026 +0000 UTC m=+580.969125822" watchObservedRunningTime="2026-11-02 21:19:01.423186464 +0000 UTC m=+580.971436244" Nov 02 21:19:01.605257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:01.605213 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:02.607300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:02.607269 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:02.607844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:02.607813 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:03.605255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:03.605220 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:03.605590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:03.605560 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:04.629976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:04.629810 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:04.629976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:04.629879 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:05.634100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:05.634058 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:05.634737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:05.634687 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:06.605852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:06.605822 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:06.606067 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:06.605876 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:07.605761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:07.605726 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-bd2cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:19:07.606248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:07.605785 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:19:08.856767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:08.856736 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:08.857284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:08.857188 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:08.857906 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:08.857885 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:19:25.943282 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:19:25.943247 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:19:43.845824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.845789 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:19:43.849587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.849565 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.946566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.946533 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:19:43.968303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968272 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968307 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968333 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968483 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968520 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968549 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968627 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968651 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbs7\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.968705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.968680 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:43.990905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:43.990879 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f4fe932-openshift-default-dockercfg-tmgdg\"" Nov 02 21:19:44.069393 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069358 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069393 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069399 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069623 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069518 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069683 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069746 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069718 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbs7\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069752 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069781 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069815 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.069920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069852 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.070045 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.069982 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.070045 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.070005 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.070164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.070138 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.070287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.070266 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.070540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.070519 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.071988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.071967 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.072141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.072123 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.122662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.122610 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.122775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.122755 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbs7\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\") pod \"e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.162758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.162737 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:44.165987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.165966 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:19:44.169943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.169926 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.218507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.218484 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-170b4fea-openshift-default-dockercfg-ls724\"" Nov 02 21:19:44.227549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.227469 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:19:44.271180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271124 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271180 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271289 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271316 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271384 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gbr\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271409 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271456 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271496 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.271640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.271551 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372417 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372479 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372531 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372574 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372634 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372695 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372726 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372776 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gbr\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.372880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372804 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.373106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.372989 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.373106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.373040 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.373106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.373070 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.373246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.373113 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.373657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.373633 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.374844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.374825 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.375005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.374989 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.426732 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.426709 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.426879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.426863 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gbr\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\") pod \"e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.492027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.491995 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:44.585372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.585341 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:19:44.591924 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:44.591898 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e499c9_1d67_4017_9066_be401f8e311a.slice/crio-6151b24e63c548e2ced0a7b0f8ed387fba58fac035143a2be58adda8d339b492 WatchSource:0}: Error finding container 6151b24e63c548e2ced0a7b0f8ed387fba58fac035143a2be58adda8d339b492: Status 404 returned error can't find the container with id 6151b24e63c548e2ced0a7b0f8ed387fba58fac035143a2be58adda8d339b492 Nov 02 21:19:44.594600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.594568 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:44.594711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.594643 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:44.594711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.594689 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:44.866226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.866197 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:19:44.871712 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:44.871687 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9d20eb_253a_4cd7_9d91_b1ff66bf4b0f.slice/crio-dad9a2c8fc601857b6322c57312d6ef415a2a0ce7d60ccdd8e24ce03eb3a81f6 WatchSource:0}: Error finding container dad9a2c8fc601857b6322c57312d6ef415a2a0ce7d60ccdd8e24ce03eb3a81f6: Status 404 returned error can't find the container with id dad9a2c8fc601857b6322c57312d6ef415a2a0ce7d60ccdd8e24ce03eb3a81f6 Nov 02 21:19:44.873893 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.873866 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:44.873980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.873965 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:44.874020 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:44.874009 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:45.546743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.546699 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerStarted","Data":"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7"} Nov 02 21:19:45.546916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.546755 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerStarted","Data":"6151b24e63c548e2ced0a7b0f8ed387fba58fac035143a2be58adda8d339b492"} Nov 02 21:19:45.548356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.548332 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerStarted","Data":"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6"} Nov 02 21:19:45.548488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.548372 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerStarted","Data":"dad9a2c8fc601857b6322c57312d6ef415a2a0ce7d60ccdd8e24ce03eb3a81f6"} Nov 02 21:19:45.657815 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.657770 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podStartSLOduration=2.657756585 podStartE2EDuration="2.657756585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:45.656428815 +0000 UTC m=+625.204678647" watchObservedRunningTime="2026-11-02 21:19:45.657756585 +0000 UTC m=+625.206006365" Nov 02 21:19:45.771009 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:45.770960 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podStartSLOduration=1.770936721 podStartE2EDuration="1.770936721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:45.768814268 +0000 UTC m=+625.317064072" watchObservedRunningTime="2026-11-02 21:19:45.770936721 +0000 UTC m=+625.319186557" Nov 02 21:19:46.163713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.163684 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:46.168339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.168318 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:46.255099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.255073 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:19:46.259072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.259051 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.327537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.327505 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:19:46.361370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.361347 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84de7f3a-openshift-default-dockercfg-r7lq9\"" Nov 02 21:19:46.392689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.392664 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.392842 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.392820 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.392901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.392870 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.392994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.392947 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdc8\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.393064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.393045 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.393115 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.393089 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.393157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.393144 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.393222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.393207 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.393254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.393232 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.493014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.492923 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:46.494362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494334 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494379 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494477 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494599 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494627 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdc8\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494655 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494688 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494754 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494781 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.494984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.494751 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.495097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.495053 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.495195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.495169 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.495375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.495350 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.497282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.497248 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.497682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.497665 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.498680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.498665 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:46.547941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.547909 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.548093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.548032 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdc8\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\") pod \"e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.554491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.554447 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:46.554609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.554496 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:46.555052 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.555032 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:19:46.555172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.555081 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:19:46.571507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.571489 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:46.989304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.989273 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:19:46.995202 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:46.995172 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd03a8e_df8e_4c92_8285_aa2707926346.slice/crio-6aa326eb44e65b77f9a77dc716b820dba06c9f3aa3cb4e7de33e2c1cc9f234ed WatchSource:0}: Error finding container 6aa326eb44e65b77f9a77dc716b820dba06c9f3aa3cb4e7de33e2c1cc9f234ed: Status 404 returned error can't find the container with id 6aa326eb44e65b77f9a77dc716b820dba06c9f3aa3cb4e7de33e2c1cc9f234ed Nov 02 21:19:46.997514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.997484 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:46.997591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.997545 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:46.997591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:46.997575 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:47.561037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:47.560998 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerStarted","Data":"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29"} Nov 02 21:19:47.561390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:47.561052 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerStarted","Data":"6aa326eb44e65b77f9a77dc716b820dba06c9f3aa3cb4e7de33e2c1cc9f234ed"} Nov 02 21:19:47.572101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:47.572074 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:47.674097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:47.674055 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podStartSLOduration=1.674042788 podStartE2EDuration="1.674042788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:47.671435886 +0000 UTC m=+627.219685680" watchObservedRunningTime="2026-11-02 21:19:47.674042788 +0000 UTC m=+627.222292568" Nov 02 21:19:48.576549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:48.576522 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:49.570659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:49.570628 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:49.571443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:49.571426 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:19:52.524904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.524865 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:19:52.528999 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.528978 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.559224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.559200 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.559375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.559354 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpzz\" (UniqueName: \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.582122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.582099 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:19:52.626171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.626147 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5478da31\"" Nov 02 21:19:52.626171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.626151 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5478da31-dockercfg-zwmth\"" Nov 02 21:19:52.660061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.660030 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpzz\" (UniqueName: \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.660165 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.660130 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.661149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.661130 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.713390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.713367 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpzz\" (UniqueName: \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\") pod \"payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:52.842544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:52.842510 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:53.217798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:53.217733 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:19:53.223686 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:53.223662 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034b46a3_64b0_46aa_a357_d8ef0e592ac5.slice/crio-76c2d4fe27a74c4215d29e39d9ffe42f01775b18c52a38254a97167dd81fd540 WatchSource:0}: Error finding container 76c2d4fe27a74c4215d29e39d9ffe42f01775b18c52a38254a97167dd81fd540: Status 404 returned error can't find the container with id 76c2d4fe27a74c4215d29e39d9ffe42f01775b18c52a38254a97167dd81fd540 Nov 02 21:19:53.594010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:53.593980 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" event={"ID":"034b46a3-64b0-46aa-a357-d8ef0e592ac5","Type":"ContainerStarted","Data":"76c2d4fe27a74c4215d29e39d9ffe42f01775b18c52a38254a97167dd81fd540"} Nov 02 21:19:54.399107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.399079 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:19:54.402623 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.402600 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.451524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.451494 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:19:54.478154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.478123 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.478304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.478182 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.478304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.478218 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.478304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.478251 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5zh\" (UniqueName: \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.498245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.498221 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-170b4fea-serving-cert\"" Nov 02 21:19:54.498372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.498279 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:19:54.498372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.498294 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:19:54.498372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.498339 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 21:19:54.502351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.502325 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:19:54.579820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.579786 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.579987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.579852 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5zh\" (UniqueName: \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.580071 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.580037 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.580149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.580126 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.580741 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:19:54.580604 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-serving-cert: secret "maas-api-e2e-ait-170b4fea-serving-cert" not found Nov 02 21:19:54.580741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.580619 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.580741 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:19:54.580690 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls podName:322110b7-cfa2-45ab-a122-763b6514f9ca nodeName:}" failed. No retries permitted until 2026-11-02 21:19:55.080669951 +0000 UTC m=+634.628919730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls") pod "maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" (UID: "322110b7-cfa2-45ab-a122-763b6514f9ca") : secret "maas-api-e2e-ait-170b4fea-serving-cert" not found Nov 02 21:19:54.583700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.583617 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:54.640849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:54.640817 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5zh\" (UniqueName: \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:55.086376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:55.086319 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:55.089135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:55.089110 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") pod \"maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:55.321938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:55.321905 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:19:55.702854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:55.702771 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:19:55.708733 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:19:55.708702 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322110b7_cfa2_45ab_a122_763b6514f9ca.slice/crio-00dce442d92446ee4d1ea3c424a427cc6aaf159936e54f2afb98a45a66fe9df1 WatchSource:0}: Error finding container 00dce442d92446ee4d1ea3c424a427cc6aaf159936e54f2afb98a45a66fe9df1: Status 404 returned error can't find the container with id 00dce442d92446ee4d1ea3c424a427cc6aaf159936e54f2afb98a45a66fe9df1 Nov 02 21:19:55.946987 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:19:55.945847 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:19:56.612563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:56.612531 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" event={"ID":"034b46a3-64b0-46aa-a357-d8ef0e592ac5","Type":"ContainerStarted","Data":"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a"} Nov 02 21:19:56.612676 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:56.612599 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:19:56.613802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:56.613779 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" event={"ID":"322110b7-cfa2-45ab-a122-763b6514f9ca","Type":"ContainerStarted","Data":"00dce442d92446ee4d1ea3c424a427cc6aaf159936e54f2afb98a45a66fe9df1"} Nov 02 21:19:56.719921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:56.719874 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" podStartSLOduration=1.8567113100000001 podStartE2EDuration="4.71986042s" totalImagesPullingTime="2.86314911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:56.718637669 +0000 UTC m=+636.266887470" watchObservedRunningTime="2026-11-02 21:19:56.71986042 +0000 UTC m=+636.268110200" Nov 02 21:19:57.620561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:57.620525 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" event={"ID":"322110b7-cfa2-45ab-a122-763b6514f9ca","Type":"ContainerStarted","Data":"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59"} Nov 02 21:19:57.731363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:57.731320 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" podStartSLOduration=2.8384132319999997 podStartE2EDuration="3.731307717s" totalImagesPullingTime="892.894485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:57.729570244 +0000 UTC m=+637.277820025" watchObservedRunningTime="2026-11-02 21:19:57.731307717 +0000 UTC m=+637.279557496" Nov 02 21:19:58.625856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:19:58.625821 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:20:04.636330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:04.636259 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:20:07.623329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:07.623302 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:20:10.562000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.561964 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:20:10.565854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.565830 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.622889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.622860 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-54a47302\"" Nov 02 21:20:10.665552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.665508 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.665741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.665712 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2mz\" (UniqueName: \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.669122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.669099 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-54a47302-dockercfg-5ckvb\"" Nov 02 21:20:10.672768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.672744 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:20:10.732266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.732234 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:20:10.736168 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.736149 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:10.767220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.767200 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2mz\" (UniqueName: \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.767329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.767264 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.767931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.767912 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.782880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.782858 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:20:10.831822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.831797 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2mz\" (UniqueName: \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\") pod \"payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.868303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.868280 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:10.868431 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.868316 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2thh\" (UniqueName: \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:10.878354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.878336 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:10.970178 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.969831 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:10.970178 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.969871 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2thh\" (UniqueName: \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:10.970669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:10.970651 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:11.032512 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.032479 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2thh\" (UniqueName: \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\") pod \"payload-processing-e2e-derive-54a47302-778578d98c-dcq26\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:11.049014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.048993 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:11.268659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.268620 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:20:11.273988 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:20:11.273959 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59a39e2_2d69_403f_8cb6_923897a84ab4.slice/crio-3eee57f6d2c080e6b150d55973b1ea496bfe809d32e903a8a00241dacb0c2246 WatchSource:0}: Error finding container 3eee57f6d2c080e6b150d55973b1ea496bfe809d32e903a8a00241dacb0c2246: Status 404 returned error can't find the container with id 3eee57f6d2c080e6b150d55973b1ea496bfe809d32e903a8a00241dacb0c2246 Nov 02 21:20:11.292531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.292509 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:20:11.297789 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.297772 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.347174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.347154 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f4fe932-serving-cert\"" Nov 02 21:20:11.376567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.376534 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:20:11.438706 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:20:11.438681 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ef207b_2d41_4bde_80a9_d022ac11185f.slice/crio-d5e0617d71b1cfdf15a5e532cfad0bb3646e75c089257c1ba358dcf4ec5a8a1e WatchSource:0}: Error finding container d5e0617d71b1cfdf15a5e532cfad0bb3646e75c089257c1ba358dcf4ec5a8a1e: Status 404 returned error can't find the container with id d5e0617d71b1cfdf15a5e532cfad0bb3646e75c089257c1ba358dcf4ec5a8a1e Nov 02 21:20:11.439831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.439807 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:20:11.476062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.476035 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.476177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.476159 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt65\" (UniqueName: \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.476274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.476261 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.476334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.476312 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.577072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.577045 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.577410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.577092 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.577410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.577157 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.577410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.577208 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt65\" (UniqueName: \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.577851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.577833 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.579361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.579343 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.579441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.579423 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.631942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.631874 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt65\" (UniqueName: \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\") pod \"maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.655535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.655512 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:11.694494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.694457 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" event={"ID":"e59a39e2-2d69-403f-8cb6-923897a84ab4","Type":"ContainerStarted","Data":"1645e302a162a50c3863f04edaeddfbbfa161c165c3213066e3b2d9cb4b7c252"} Nov 02 21:20:11.694632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.694518 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" event={"ID":"e59a39e2-2d69-403f-8cb6-923897a84ab4","Type":"ContainerStarted","Data":"3eee57f6d2c080e6b150d55973b1ea496bfe809d32e903a8a00241dacb0c2246"} Nov 02 21:20:11.694632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.694567 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:11.696483 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.696309 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" event={"ID":"62ef207b-2d41-4bde-80a9-d022ac11185f","Type":"ContainerStarted","Data":"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5"} Nov 02 21:20:11.696483 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.696362 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" event={"ID":"62ef207b-2d41-4bde-80a9-d022ac11185f","Type":"ContainerStarted","Data":"d5e0617d71b1cfdf15a5e532cfad0bb3646e75c089257c1ba358dcf4ec5a8a1e"} Nov 02 21:20:11.696483 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.696394 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:11.940554 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.940459 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" podStartSLOduration=1.940443975 podStartE2EDuration="1.940443975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:11.939223431 +0000 UTC m=+651.487473214" watchObservedRunningTime="2026-11-02 21:20:11.940443975 +0000 UTC m=+651.488693755" Nov 02 21:20:11.940912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:11.940859 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" podStartSLOduration=1.940847669 podStartE2EDuration="1.940847669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:11.818882671 +0000 UTC m=+651.367132453" watchObservedRunningTime="2026-11-02 21:20:11.940847669 +0000 UTC m=+651.489097452" Nov 02 21:20:12.108978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:12.107619 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:20:12.133226 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:20:12.133199 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e26e48e_b331_4ffd_a3b2_ca562026d699.slice/crio-5ad0b2a58fc82c23378e8fb538c9009d7427a53298c7ff48779c4d6dbfbdb297 WatchSource:0}: Error finding container 5ad0b2a58fc82c23378e8fb538c9009d7427a53298c7ff48779c4d6dbfbdb297: Status 404 returned error can't find the container with id 5ad0b2a58fc82c23378e8fb538c9009d7427a53298c7ff48779c4d6dbfbdb297 Nov 02 21:20:12.702667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:12.702627 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" event={"ID":"4e26e48e-b331-4ffd-a3b2-ca562026d699","Type":"ContainerStarted","Data":"5ad0b2a58fc82c23378e8fb538c9009d7427a53298c7ff48779c4d6dbfbdb297"} Nov 02 21:20:13.710568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:13.710535 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" event={"ID":"4e26e48e-b331-4ffd-a3b2-ca562026d699","Type":"ContainerStarted","Data":"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e"} Nov 02 21:20:13.711020 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:13.710651 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:13.817505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:13.817454 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" podStartSLOduration=1.954901319 podStartE2EDuration="2.817438892s" totalImagesPullingTime="862.537573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:13.817271297 +0000 UTC m=+653.365521078" watchObservedRunningTime="2026-11-02 21:20:13.817438892 +0000 UTC m=+653.365688672" Nov 02 21:20:16.749382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:16.748532 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:53226->10.134.0.43:15021: read: connection reset by peer" start-of-body= Nov 02 21:20:16.749382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:16.748588 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:53226->10.134.0.43:15021: read: connection reset by peer" Nov 02 21:20:17.734869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.734832 5859 generic.go:356] "Generic (PLEG): container finished" podID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerID="5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6" exitCode=0 Nov 02 21:20:17.735063 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.734907 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerDied","Data":"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6"} Nov 02 21:20:17.735278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.735265 5859 scope.go:122] "RemoveContainer" containerID="5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6" Nov 02 21:20:17.736586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.736560 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:17.736646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.736633 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:17.736681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:17.736667 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:18.758272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:18.758232 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerStarted","Data":"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb"} Nov 02 21:20:19.493216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:19.493183 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:20:19.493433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:19.493412 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:20:19.493482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:19.493460 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 21:20:19.744023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:19.743135 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:20:20.492766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:20.492732 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:20:20.493179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:20.492794 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 21:20:21.492533 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:21.492500 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:20:21.492718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:21.492550 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 21:20:22.493072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:22.493039 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:20:22.493457 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:22.493091 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 21:20:22.739976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:22.739129 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:20:22.741985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:22.740266 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:20:23.492375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:23.492345 5859 patch_prober.go:29] interesting pod/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 21:20:23.492522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:23.492404 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 21:20:24.496673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:24.496643 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:20:24.497074 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:24.497051 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:20:24.497875 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:24.497856 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:20:25.943319 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:20:25.943274 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:20:26.810134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.810056 5859 generic.go:356] "Generic (PLEG): container finished" podID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerID="ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29" exitCode=0 Nov 02 21:20:26.810134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.810115 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerDied","Data":"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29"} Nov 02 21:20:26.810548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.810526 5859 scope.go:122] "RemoveContainer" containerID="ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29" Nov 02 21:20:26.811801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.811770 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:26.811900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.811852 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:26.811976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:26.811900 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:27.833868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:27.833832 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerStarted","Data":"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0"} Nov 02 21:20:28.572104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:28.572076 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:20:28.572347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:28.572284 5859 patch_prober.go:29] interesting pod/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:20:28.572469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:28.572451 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 21:20:29.572743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:29.572715 5859 patch_prober.go:29] interesting pod/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:20:29.573208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:29.573179 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 21:20:30.572138 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:30.572104 5859 patch_prober.go:29] interesting pod/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:20:30.572310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:30.572155 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 21:20:31.572048 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:31.572001 5859 patch_prober.go:29] interesting pod/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:20:31.572561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:31.572077 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 21:20:32.572904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:32.572872 5859 patch_prober.go:29] interesting pod/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 21:20:32.573305 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:32.572947 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 21:20:33.576836 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:33.576811 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:20:33.577221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:33.577199 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:20:33.577828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:33.577807 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:20:36.901636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.901552 5859 generic.go:356] "Generic (PLEG): container finished" podID="11e499c9-1d67-4017-9066-be401f8e311a" containerID="0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7" exitCode=0 Nov 02 21:20:36.902164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.901626 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerDied","Data":"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7"} Nov 02 21:20:36.902164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.902133 5859 scope.go:122] "RemoveContainer" containerID="0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7" Nov 02 21:20:36.903323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.903298 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:36.903418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.903383 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:36.903455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:36.903416 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:37.932133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:37.932098 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerStarted","Data":"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954"} Nov 02 21:20:38.163683 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:38.163651 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:20:38.163870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:38.163836 5859 patch_prober.go:29] interesting pod/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:20:38.163912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:38.163890 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 21:20:39.163353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:39.163316 5859 patch_prober.go:29] interesting pod/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:20:39.163768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:39.163372 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 21:20:40.163164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:40.163124 5859 patch_prober.go:29] interesting pod/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:20:40.163380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:40.163191 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 21:20:41.163252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:41.163214 5859 patch_prober.go:29] interesting pod/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:20:41.163461 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:41.163282 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 21:20:42.164124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:42.164078 5859 patch_prober.go:29] interesting pod/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 21:20:42.164628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:42.164139 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 21:20:43.167019 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:43.166990 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:20:43.167466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:43.167377 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:20:43.168087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:43.168067 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:20:51.072873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:51.072837 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:20:55.943409 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:20:55.943375 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:20:56.073520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.073489 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" podUID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" containerName="payload-processing" containerID="cri-o://cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a" gracePeriod=25 Nov 02 21:20:56.321283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.321259 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:20:56.405271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.405248 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\") pod \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " Nov 02 21:20:56.405408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.405294 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\" (UniqueName: \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\") pod \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\" (UID: \"034b46a3-64b0-46aa-a357-d8ef0e592ac5\") " Nov 02 21:20:56.405610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.405587 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume" pod "034b46a3-64b0-46aa-a357-d8ef0e592ac5" (UID: "034b46a3-64b0-46aa-a357-d8ef0e592ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:20:56.407310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.407290 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz" pod "034b46a3-64b0-46aa-a357-d8ef0e592ac5" (UID: "034b46a3-64b0-46aa-a357-d8ef0e592ac5"). InnerVolumeSpecName "kube-api-access-xzpzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:56.506418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.506363 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/034b46a3-64b0-46aa-a357-d8ef0e592ac5-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:56.506418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:56.506385 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzpzz\" (UniqueName: \"kubernetes.io/projected/034b46a3-64b0-46aa-a357-d8ef0e592ac5-kube-api-access-xzpzz\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:57.037355 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.037323 5859 generic.go:356] "Generic (PLEG): container finished" podID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" containerID="cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a" exitCode=0 Nov 02 21:20:57.037827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.037378 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" Nov 02 21:20:57.037827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.037412 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" event={"ID":"034b46a3-64b0-46aa-a357-d8ef0e592ac5","Type":"ContainerDied","Data":"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a"} Nov 02 21:20:57.037827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.037472 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6" event={"ID":"034b46a3-64b0-46aa-a357-d8ef0e592ac5","Type":"ContainerDied","Data":"76c2d4fe27a74c4215d29e39d9ffe42f01775b18c52a38254a97167dd81fd540"} Nov 02 21:20:57.037827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.037507 5859 scope.go:122] "RemoveContainer" containerID="cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a" Nov 02 21:20:57.046489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.046462 5859 scope.go:122] "RemoveContainer" containerID="cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a" Nov 02 21:20:57.046738 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:20:57.046721 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a\": container with ID starting with cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a not found: ID does not exist" containerID="cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a" Nov 02 21:20:57.046785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.046746 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a"} err="failed to get container status \"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a\": rpc error: code = NotFound desc = could not find container \"cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a\": container with ID starting with cdd2673ddcdab6f9e6fb0f447671e61d320796a28b2f0e454414f4789f17d09a not found: ID does not exist" Nov 02 21:20:57.191220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.191194 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:20:57.237628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:57.237601 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5478da31-56b4d57d79-29pz6"] Nov 02 21:20:58.420379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.420348 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:20:58.420747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.420538 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" podUID="322110b7-cfa2-45ab-a122-763b6514f9ca" containerName="maas-api" containerID="cri-o://c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59" gracePeriod=30 Nov 02 21:20:58.571370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.571340 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" path="/var/lib/kubelet/pods/034b46a3-64b0-46aa-a357-d8ef0e592ac5/volumes" Nov 02 21:20:58.701910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.701834 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:20:58.728657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.728631 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\") pod \"322110b7-cfa2-45ab-a122-763b6514f9ca\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " Nov 02 21:20:58.728792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.728692 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") pod \"322110b7-cfa2-45ab-a122-763b6514f9ca\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " Nov 02 21:20:58.728792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.728742 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\") pod \"322110b7-cfa2-45ab-a122-763b6514f9ca\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " Nov 02 21:20:58.728864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.728801 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\" (UniqueName: \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\") pod \"322110b7-cfa2-45ab-a122-763b6514f9ca\" (UID: \"322110b7-cfa2-45ab-a122-763b6514f9ca\") " Nov 02 21:20:58.729241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.729209 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca" pod "322110b7-cfa2-45ab-a122-763b6514f9ca" (UID: "322110b7-cfa2-45ab-a122-763b6514f9ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:20:58.730910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.730884 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls" pod "322110b7-cfa2-45ab-a122-763b6514f9ca" (UID: "322110b7-cfa2-45ab-a122-763b6514f9ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:58.731019 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.730912 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh" pod "322110b7-cfa2-45ab-a122-763b6514f9ca" (UID: "322110b7-cfa2-45ab-a122-763b6514f9ca"). InnerVolumeSpecName "kube-api-access-pg5zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:58.731019 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.730927 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs" pod "322110b7-cfa2-45ab-a122-763b6514f9ca" (UID: "322110b7-cfa2-45ab-a122-763b6514f9ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:58.830589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.830558 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322110b7-cfa2-45ab-a122-763b6514f9ca-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:58.830589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.830581 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg5zh\" (UniqueName: \"kubernetes.io/projected/322110b7-cfa2-45ab-a122-763b6514f9ca-kube-api-access-pg5zh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:58.830589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.830591 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:58.830825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:58.830599 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322110b7-cfa2-45ab-a122-763b6514f9ca-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:59.051665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.051587 5859 generic.go:356] "Generic (PLEG): container finished" podID="322110b7-cfa2-45ab-a122-763b6514f9ca" containerID="c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59" exitCode=0 Nov 02 21:20:59.051665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.051633 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" Nov 02 21:20:59.051846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.051675 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" event={"ID":"322110b7-cfa2-45ab-a122-763b6514f9ca","Type":"ContainerDied","Data":"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59"} Nov 02 21:20:59.051846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.051725 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29" event={"ID":"322110b7-cfa2-45ab-a122-763b6514f9ca","Type":"ContainerDied","Data":"00dce442d92446ee4d1ea3c424a427cc6aaf159936e54f2afb98a45a66fe9df1"} Nov 02 21:20:59.051846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.051747 5859 scope.go:122] "RemoveContainer" containerID="c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59" Nov 02 21:20:59.061667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.061653 5859 scope.go:122] "RemoveContainer" containerID="c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59" Nov 02 21:20:59.061940 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:20:59.061919 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59\": container with ID starting with c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59 not found: ID does not exist" containerID="c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59" Nov 02 21:20:59.062031 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.061963 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59"} err="failed to get container status \"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59\": rpc error: code = NotFound desc = could not find container \"c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59\": container with ID starting with c2374cec5ab08b67607c99016629b75558d4587287ebc3b97721b6f172854e59 not found: ID does not exist" Nov 02 21:20:59.204921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.204897 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:20:59.208223 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:20:59.208202 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-170b4fea-85f9894fdf-rbm29"] Nov 02 21:21:00.570123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:00.570088 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322110b7-cfa2-45ab-a122-763b6514f9ca" path="/var/lib/kubelet/pods/322110b7-cfa2-45ab-a122-763b6514f9ca/volumes" Nov 02 21:21:02.248458 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:02.248423 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:21:02.248907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:02.248647 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" containerID="cri-o://e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb" gracePeriod=30 Nov 02 21:21:04.463094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:04.463063 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:21:04.463595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:04.463324 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" containerID="cri-o://11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0" gracePeriod=30 Nov 02 21:21:05.837008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.836947 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 21:21:05.837582 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837546 5859 cpu_manager.go:414] "containerMap: removing container" podUID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" containerName="payload-processing" Nov 02 21:21:05.837582 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837567 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" containerName="payload-processing" Nov 02 21:21:05.837698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837589 5859 cpu_manager.go:414] "containerMap: removing container" podUID="322110b7-cfa2-45ab-a122-763b6514f9ca" containerName="maas-api" Nov 02 21:21:05.837698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837597 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322110b7-cfa2-45ab-a122-763b6514f9ca" containerName="maas-api" Nov 02 21:21:05.837781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837771 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="322110b7-cfa2-45ab-a122-763b6514f9ca" containerName="maas-api" Nov 02 21:21:05.837829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.837786 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="034b46a3-64b0-46aa-a357-d8ef0e592ac5" containerName="payload-processing" Nov 02 21:21:05.841068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.841048 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.892685 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.892654 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\"" Nov 02 21:21:05.913460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.913353 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.913460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.913389 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.913902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.913867 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.913998 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.914096 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4lh\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.914147 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.914210 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914281 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.914244 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.914334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.914309 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:05.941920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:05.941892 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 21:21:06.015299 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015260 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4lh\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015332 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015388 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015423 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015463 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015512 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015536 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015602 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015656 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015885 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.015983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015908 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.016094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.015997 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.016226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.016183 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.016226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.016186 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.017988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.017939 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.018231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.018214 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.072441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.072409 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.072638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.072619 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4lh\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.154361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.154279 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:06.576221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.576192 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 21:21:06.621644 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:06.621616 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcffb9cf_bd01_4a48_9972_eaf53413e548.slice/crio-61eba1bec86bb6f08347838ef0a46da0f52a30b0c1665d4a510ad34caaad9bad WatchSource:0}: Error finding container 61eba1bec86bb6f08347838ef0a46da0f52a30b0c1665d4a510ad34caaad9bad: Status 404 returned error can't find the container with id 61eba1bec86bb6f08347838ef0a46da0f52a30b0c1665d4a510ad34caaad9bad Nov 02 21:21:06.624313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.624276 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:06.624398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.624350 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:06.624398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:06.624380 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:07.101674 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.101632 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerStarted","Data":"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b"} Nov 02 21:21:07.102134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.101704 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerStarted","Data":"61eba1bec86bb6f08347838ef0a46da0f52a30b0c1665d4a510ad34caaad9bad"} Nov 02 21:21:07.154646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.154617 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:07.159622 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.159595 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:07.211780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.211729 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podStartSLOduration=2.21171622 podStartE2EDuration="2.21171622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:07.211242381 +0000 UTC m=+706.759492208" watchObservedRunningTime="2026-11-02 21:21:07.21171622 +0000 UTC m=+706.759966000" Nov 02 21:21:07.531644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.531622 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:21:07.635896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.635820 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.635896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.635859 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.635896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.635877 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636021 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636095 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636133 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636187 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636205 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:07.636399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636281 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:07.636399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636318 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636364 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\") pod \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\" (UID: \"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f\") " Nov 02 21:21:07.636604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636400 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:07.636604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636493 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:07.636705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636660 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:07.636822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636797 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.636822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636825 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.637037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636839 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.637037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636855 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.637037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.636871 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.638185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.638162 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:07.638467 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.638446 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:21:07.638587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.638542 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:07.638587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.638550 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr" pod "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" (UID: "5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f"). InnerVolumeSpecName "kube-api-access-42gbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:07.738046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.738013 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.738046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.738040 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.738046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.738050 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:07.738265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:07.738060 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gbr\" (UniqueName: \"kubernetes.io/projected/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f-kube-api-access-42gbr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:08.109704 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.109669 5859 generic.go:356] "Generic (PLEG): container finished" podID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerID="e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb" exitCode=0 Nov 02 21:21:08.110155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.109717 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" Nov 02 21:21:08.110155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.109749 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerDied","Data":"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb"} Nov 02 21:21:08.110155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.109800 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq" event={"ID":"5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f","Type":"ContainerDied","Data":"dad9a2c8fc601857b6322c57312d6ef415a2a0ce7d60ccdd8e24ce03eb3a81f6"} Nov 02 21:21:08.110155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.109823 5859 scope.go:122] "RemoveContainer" containerID="e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb" Nov 02 21:21:08.110357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.110273 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:08.111367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.111346 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:08.122785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.122767 5859 scope.go:122] "RemoveContainer" containerID="5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6" Nov 02 21:21:08.131249 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.131231 5859 scope.go:122] "RemoveContainer" containerID="e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb" Nov 02 21:21:08.131523 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:08.131503 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb\": container with ID starting with e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb not found: ID does not exist" containerID="e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb" Nov 02 21:21:08.131615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.131527 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb"} err="failed to get container status \"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb\": rpc error: code = NotFound desc = could not find container \"e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb\": container with ID starting with e0722d00162ed17e2f8b58a4c5e186d18972841f438ef36133e28a49eb5f09eb not found: ID does not exist" Nov 02 21:21:08.131615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.131545 5859 scope.go:122] "RemoveContainer" containerID="5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6" Nov 02 21:21:08.131765 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:08.131748 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6\": container with ID starting with 5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6 not found: ID does not exist" containerID="5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6" Nov 02 21:21:08.131808 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.131771 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6"} err="failed to get container status \"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6\": rpc error: code = NotFound desc = could not find container \"5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6\": container with ID starting with 5ad2f80a563bff93cc83b2835de2810c0cf3d5cabb0fb576b854095e3eecc5f6 not found: ID does not exist" Nov 02 21:21:08.310650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.310615 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:21:08.357025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.356992 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-170b4fea-openshift-default-8579f78765-hdgvq"] Nov 02 21:21:08.570924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:08.570892 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" path="/var/lib/kubelet/pods/5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f/volumes" Nov 02 21:21:09.720557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.720533 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:21:09.857708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857682 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.857830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857718 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.857830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857755 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.857830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857789 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857840 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857866 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857902 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.857972 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:09.858214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858062 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858114 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:09.858214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858150 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\") pod \"dfd03a8e-df8e-4c92-8285-aa2707926346\" (UID: \"dfd03a8e-df8e-4c92-8285-aa2707926346\") " Nov 02 21:21:09.858360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858251 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:09.858414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858386 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:09.858591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858573 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:09.858668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858593 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.858668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858615 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.858668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858630 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.858668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.858645 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.859909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.859891 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:09.860218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.860201 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:09.860383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.860358 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:21:09.860476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.860392 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8" pod "dfd03a8e-df8e-4c92-8285-aa2707926346" (UID: "dfd03a8e-df8e-4c92-8285-aa2707926346"). InnerVolumeSpecName "kube-api-access-nmdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:09.959970 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.959931 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.959970 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.959974 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.960119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.959983 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd03a8e-df8e-4c92-8285-aa2707926346-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.960119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.959992 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdc8\" (UniqueName: \"kubernetes.io/projected/dfd03a8e-df8e-4c92-8285-aa2707926346-kube-api-access-nmdc8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:09.960119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:09.960003 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd03a8e-df8e-4c92-8285-aa2707926346-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:10.123900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.123827 5859 generic.go:356] "Generic (PLEG): container finished" podID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerID="11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0" exitCode=0 Nov 02 21:21:10.123900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.123886 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" Nov 02 21:21:10.124120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.123909 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerDied","Data":"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0"} Nov 02 21:21:10.124120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.123992 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz" event={"ID":"dfd03a8e-df8e-4c92-8285-aa2707926346","Type":"ContainerDied","Data":"6aa326eb44e65b77f9a77dc716b820dba06c9f3aa3cb4e7de33e2c1cc9f234ed"} Nov 02 21:21:10.124120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.124017 5859 scope.go:122] "RemoveContainer" containerID="11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0" Nov 02 21:21:10.134237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.134218 5859 scope.go:122] "RemoveContainer" containerID="ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29" Nov 02 21:21:10.142261 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.142243 5859 scope.go:122] "RemoveContainer" containerID="11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0" Nov 02 21:21:10.142517 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:10.142501 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0\": container with ID starting with 11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0 not found: ID does not exist" containerID="11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0" Nov 02 21:21:10.142564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.142526 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0"} err="failed to get container status \"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0\": rpc error: code = NotFound desc = could not find container \"11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0\": container with ID starting with 11cef625e9993742ebf30f1d90cb5e6c31c63a9eb23047ca4efedf3d445710e0 not found: ID does not exist" Nov 02 21:21:10.142564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.142543 5859 scope.go:122] "RemoveContainer" containerID="ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29" Nov 02 21:21:10.142759 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:10.142741 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29\": container with ID starting with ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29 not found: ID does not exist" containerID="ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29" Nov 02 21:21:10.142794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.142766 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29"} err="failed to get container status \"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29\": rpc error: code = NotFound desc = could not find container \"ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29\": container with ID starting with ce7698b5c91a779f6646274bb22c82984e1fc6274816c81d562292f0ec1a7a29 not found: ID does not exist" Nov 02 21:21:10.277473 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.277443 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:21:10.283132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.283111 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84de7f3a-openshift-default-5fbdf85f7-v72pz"] Nov 02 21:21:10.570166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:10.570133 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" path="/var/lib/kubelet/pods/dfd03a8e-df8e-4c92-8285-aa2707926346/volumes" Nov 02 21:21:11.334680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.334643 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 21:21:11.335693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.335668 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:11.335693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.335694 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:11.335842 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.335718 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:11.335842 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.335726 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:11.336030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.336016 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:11.336094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.336037 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:11.336094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.336062 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:11.342109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.342085 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.391290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.391260 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 21:21:11.393522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.393498 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f51151-dockercfg-dm9gr\"" Nov 02 21:21:11.393687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.393663 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f51151\"" Nov 02 21:21:11.474730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.474691 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.474730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.474730 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwrb\" (UniqueName: \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.575849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.575818 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.576037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.575857 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwrb\" (UniqueName: \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.576462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.576443 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.630508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.630448 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwrb\" (UniqueName: \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\") pod \"payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:11.654307 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:11.654285 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:12.017613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:12.017548 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 21:21:12.024198 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:12.024173 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf47427_58b4_4e55_8258_b47a87a21013.slice/crio-94f424f01c8a786b9e7d65fff233a9f398aa76b76dbd9ca1887ff23821fc74c7 WatchSource:0}: Error finding container 94f424f01c8a786b9e7d65fff233a9f398aa76b76dbd9ca1887ff23821fc74c7: Status 404 returned error can't find the container with id 94f424f01c8a786b9e7d65fff233a9f398aa76b76dbd9ca1887ff23821fc74c7 Nov 02 21:21:12.139136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:12.139111 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" event={"ID":"abf47427-58b4-4e55-8258-b47a87a21013","Type":"ContainerStarted","Data":"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0"} Nov 02 21:21:12.139235 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:12.139157 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" event={"ID":"abf47427-58b4-4e55-8258-b47a87a21013","Type":"ContainerStarted","Data":"94f424f01c8a786b9e7d65fff233a9f398aa76b76dbd9ca1887ff23821fc74c7"} Nov 02 21:21:12.139235 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:12.139186 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:12.248349 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:12.248306 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" podStartSLOduration=1.248293463 podStartE2EDuration="1.248293463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:12.246790746 +0000 UTC m=+711.795040527" watchObservedRunningTime="2026-11-02 21:21:12.248293463 +0000 UTC m=+711.796543244" Nov 02 21:21:13.873797 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:13.873765 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:21:13.931294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:13.931263 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:21:14.782047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:14.782016 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:21:14.782215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:14.782192 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" podUID="4e26e48e-b331-4ffd-a3b2-ca562026d699" containerName="maas-api" containerID="cri-o://c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e" gracePeriod=30 Nov 02 21:21:15.033668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.033612 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:21:15.119463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.119439 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\") pod \"4e26e48e-b331-4ffd-a3b2-ca562026d699\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " Nov 02 21:21:15.119593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.119504 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\") pod \"4e26e48e-b331-4ffd-a3b2-ca562026d699\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " Nov 02 21:21:15.119593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.119553 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\" (UniqueName: \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\") pod \"4e26e48e-b331-4ffd-a3b2-ca562026d699\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " Nov 02 21:21:15.119721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.119606 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\") pod \"4e26e48e-b331-4ffd-a3b2-ca562026d699\" (UID: \"4e26e48e-b331-4ffd-a3b2-ca562026d699\") " Nov 02 21:21:15.119924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.119878 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca" pod "4e26e48e-b331-4ffd-a3b2-ca562026d699" (UID: "4e26e48e-b331-4ffd-a3b2-ca562026d699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:15.121470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.121444 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls" pod "4e26e48e-b331-4ffd-a3b2-ca562026d699" (UID: "4e26e48e-b331-4ffd-a3b2-ca562026d699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:15.121601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.121578 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs" pod "4e26e48e-b331-4ffd-a3b2-ca562026d699" (UID: "4e26e48e-b331-4ffd-a3b2-ca562026d699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:15.121601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.121591 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65" pod "4e26e48e-b331-4ffd-a3b2-ca562026d699" (UID: "4e26e48e-b331-4ffd-a3b2-ca562026d699"). InnerVolumeSpecName "kube-api-access-9qt65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:15.157240 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.157214 5859 generic.go:356] "Generic (PLEG): container finished" podID="4e26e48e-b331-4ffd-a3b2-ca562026d699" containerID="c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e" exitCode=0 Nov 02 21:21:15.157362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.157261 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" Nov 02 21:21:15.157362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.157281 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" event={"ID":"4e26e48e-b331-4ffd-a3b2-ca562026d699","Type":"ContainerDied","Data":"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e"} Nov 02 21:21:15.157362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.157319 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx" event={"ID":"4e26e48e-b331-4ffd-a3b2-ca562026d699","Type":"ContainerDied","Data":"5ad0b2a58fc82c23378e8fb538c9009d7427a53298c7ff48779c4d6dbfbdb297"} Nov 02 21:21:15.157362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.157340 5859 scope.go:122] "RemoveContainer" containerID="c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e" Nov 02 21:21:15.167834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.167812 5859 scope.go:122] "RemoveContainer" containerID="c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e" Nov 02 21:21:15.168139 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:15.168113 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e\": container with ID starting with c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e not found: ID does not exist" containerID="c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e" Nov 02 21:21:15.168242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.168145 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e"} err="failed to get container status \"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e\": rpc error: code = NotFound desc = could not find container \"c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e\": container with ID starting with c6849320157133a328e1af48a1570cd803690e89d0235ff289eede344f8d2b7e not found: ID does not exist" Nov 02 21:21:15.221228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.221205 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:15.221228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.221226 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e26e48e-b331-4ffd-a3b2-ca562026d699-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:15.221359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.221236 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e26e48e-b331-4ffd-a3b2-ca562026d699-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:15.221359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.221245 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qt65\" (UniqueName: \"kubernetes.io/projected/4e26e48e-b331-4ffd-a3b2-ca562026d699-kube-api-access-9qt65\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:15.312947 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.312871 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:21:15.315693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.315670 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f4fe932-84c6f4474b-8fcsx"] Nov 02 21:21:15.879260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879227 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 21:21:15.879705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879691 5859 cpu_manager.go:414] "containerMap: removing container" podUID="4e26e48e-b331-4ffd-a3b2-ca562026d699" containerName="maas-api" Nov 02 21:21:15.879747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879706 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e26e48e-b331-4ffd-a3b2-ca562026d699" containerName="maas-api" Nov 02 21:21:15.879747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879742 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:15.879811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879746 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9d20eb-253a-4cd7-9d91-b1ff66bf4b0f" containerName="istio-proxy" Nov 02 21:21:15.879811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879754 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:15.879811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879761 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:15.879918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879838 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd03a8e-df8e-4c92-8285-aa2707926346" containerName="istio-proxy" Nov 02 21:21:15.879918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.879853 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e26e48e-b331-4ffd-a3b2-ca562026d699" containerName="maas-api" Nov 02 21:21:15.884187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.884168 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:15.938376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.938351 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d6cb38\"" Nov 02 21:21:15.938636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.938606 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-d6cb38-dockercfg-br9zx\"" Nov 02 21:21:15.952692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:15.952645 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 21:21:16.030395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.030369 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.030395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.030405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfk8\" (UniqueName: \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.045143 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.045112 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 21:21:16.049014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.048994 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.102680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.102659 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 21:21:16.131874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.131811 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.131874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.131854 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfk8\" (UniqueName: \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.132068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.131893 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267hv\" (UniqueName: \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.132068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.131991 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.132536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.132508 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.186895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.186867 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfk8\" (UniqueName: \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\") pod \"payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.197078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.197057 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:16.233121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.233095 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.233222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.233205 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267hv\" (UniqueName: \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.233722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.233700 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.286723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.286693 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-267hv\" (UniqueName: \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\") pod \"payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.362082 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.362056 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:16.562343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.562314 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 21:21:16.570552 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:16.570523 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eca70f_a55d_40a6_9e01_1504eddb3704.slice/crio-9d5619a44853ff35a71a24abd70d45a7043a76eda639dd2f83ae826023d2d425 WatchSource:0}: Error finding container 9d5619a44853ff35a71a24abd70d45a7043a76eda639dd2f83ae826023d2d425: Status 404 returned error can't find the container with id 9d5619a44853ff35a71a24abd70d45a7043a76eda639dd2f83ae826023d2d425 Nov 02 21:21:16.574455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.574428 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e26e48e-b331-4ffd-a3b2-ca562026d699" path="/var/lib/kubelet/pods/4e26e48e-b331-4ffd-a3b2-ca562026d699/volumes" Nov 02 21:21:16.726469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:16.726394 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 21:21:16.732192 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:16.732166 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb295cb13_d8bb_42e7_ac51_c87718503401.slice/crio-fa866a16bcae33c7aebb286dc56dc6bc8a285ebe86abc9278cbd88fb3d9aa525 WatchSource:0}: Error finding container fa866a16bcae33c7aebb286dc56dc6bc8a285ebe86abc9278cbd88fb3d9aa525: Status 404 returned error can't find the container with id fa866a16bcae33c7aebb286dc56dc6bc8a285ebe86abc9278cbd88fb3d9aa525 Nov 02 21:21:17.170315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.170279 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" event={"ID":"d3eca70f-a55d-40a6-9e01-1504eddb3704","Type":"ContainerStarted","Data":"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173"} Nov 02 21:21:17.170740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.170347 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:17.170740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.170369 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" event={"ID":"d3eca70f-a55d-40a6-9e01-1504eddb3704","Type":"ContainerStarted","Data":"9d5619a44853ff35a71a24abd70d45a7043a76eda639dd2f83ae826023d2d425"} Nov 02 21:21:17.171896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.171874 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" event={"ID":"b295cb13-d8bb-42e7-ac51-c87718503401","Type":"ContainerStarted","Data":"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29"} Nov 02 21:21:17.172034 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.171916 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" event={"ID":"b295cb13-d8bb-42e7-ac51-c87718503401","Type":"ContainerStarted","Data":"fa866a16bcae33c7aebb286dc56dc6bc8a285ebe86abc9278cbd88fb3d9aa525"} Nov 02 21:21:17.172034 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.171942 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:17.278672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.278622 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" podStartSLOduration=2.278609729 podStartE2EDuration="2.278609729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:17.276801459 +0000 UTC m=+716.825051266" watchObservedRunningTime="2026-11-02 21:21:17.278609729 +0000 UTC m=+716.826859509" Nov 02 21:21:17.385165 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:17.385122 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" podStartSLOduration=1.385110069 podStartE2EDuration="1.385110069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:17.382622386 +0000 UTC m=+716.930872169" watchObservedRunningTime="2026-11-02 21:21:17.385110069 +0000 UTC m=+716.933359849" Nov 02 21:21:18.179180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.179149 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 21:21:18.182915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.182893 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.279864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.279802 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0c9014\"" Nov 02 21:21:18.279864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.279804 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0c9014-dockercfg-7hk7c\"" Nov 02 21:21:18.282582 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.282554 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 21:21:18.371880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.371849 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.372084 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.372063 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tzv\" (UniqueName: \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.379054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.379027 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 21:21:18.382713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.382697 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.451119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.451060 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 21:21:18.473629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.473467 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.473629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.473523 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl54b\" (UniqueName: \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.473629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.473585 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.473858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.473724 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tzv\" (UniqueName: \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.474602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.474579 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.491693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.491670 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:21:18.491905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.491881 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" containerID="cri-o://2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954" gracePeriod=30 Nov 02 21:21:18.527506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.527477 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tzv\" (UniqueName: \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\") pod \"payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.574900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.574873 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.575026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.575008 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl54b\" (UniqueName: \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.575513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.575492 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.628730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.628700 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl54b\" (UniqueName: \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\") pod \"payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.695125 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.695098 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:18.796489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.796448 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:18.875087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.875009 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" podUID="62ef207b-2d41-4bde-80a9-d022ac11185f" containerName="payload-processing" containerID="cri-o://97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5" gracePeriod=25 Nov 02 21:21:18.931697 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:18.931667 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" podUID="e59a39e2-2d69-403f-8cb6-923897a84ab4" containerName="payload-pre-processing" containerID="cri-o://1645e302a162a50c3863f04edaeddfbbfa161c165c3213066e3b2d9cb4b7c252" gracePeriod=25 Nov 02 21:21:19.067480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.067446 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 21:21:19.074608 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:19.074583 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1801c51a_c2ac_441f_a8f0_316c35aec026.slice/crio-0fad1b8188aa6db993a9f42c8e70ab2b1082011c9148664074ca97828ccc9b58 WatchSource:0}: Error finding container 0fad1b8188aa6db993a9f42c8e70ab2b1082011c9148664074ca97828ccc9b58: Status 404 returned error can't find the container with id 0fad1b8188aa6db993a9f42c8e70ab2b1082011c9148664074ca97828ccc9b58 Nov 02 21:21:19.169429 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.169401 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 21:21:19.173465 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.173446 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:21:19.174022 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:19.173976 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd53e7b_339d_4433_900b_141ea31a8b8a.slice/crio-aa5d172c63cc352db3d3429e7b57a2b06363b37cf03f2ed6c487b0335a20a229 WatchSource:0}: Error finding container aa5d172c63cc352db3d3429e7b57a2b06363b37cf03f2ed6c487b0335a20a229: Status 404 returned error can't find the container with id aa5d172c63cc352db3d3429e7b57a2b06363b37cf03f2ed6c487b0335a20a229 Nov 02 21:21:19.186625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.186594 5859 generic.go:356] "Generic (PLEG): container finished" podID="62ef207b-2d41-4bde-80a9-d022ac11185f" containerID="97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5" exitCode=0 Nov 02 21:21:19.187161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.187141 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" event={"ID":"62ef207b-2d41-4bde-80a9-d022ac11185f","Type":"ContainerDied","Data":"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5"} Nov 02 21:21:19.187324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.187309 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" event={"ID":"62ef207b-2d41-4bde-80a9-d022ac11185f","Type":"ContainerDied","Data":"d5e0617d71b1cfdf15a5e532cfad0bb3646e75c089257c1ba358dcf4ec5a8a1e"} Nov 02 21:21:19.187447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.187436 5859 scope.go:122] "RemoveContainer" containerID="97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5" Nov 02 21:21:19.187845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.187723 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26" Nov 02 21:21:19.194202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.194173 5859 generic.go:356] "Generic (PLEG): container finished" podID="e59a39e2-2d69-403f-8cb6-923897a84ab4" containerID="1645e302a162a50c3863f04edaeddfbbfa161c165c3213066e3b2d9cb4b7c252" exitCode=0 Nov 02 21:21:19.194433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.194264 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" event={"ID":"e59a39e2-2d69-403f-8cb6-923897a84ab4","Type":"ContainerDied","Data":"1645e302a162a50c3863f04edaeddfbbfa161c165c3213066e3b2d9cb4b7c252"} Nov 02 21:21:19.196021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.195996 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" event={"ID":"1801c51a-c2ac-441f-a8f0-316c35aec026","Type":"ContainerStarted","Data":"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4"} Nov 02 21:21:19.196129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.196039 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" event={"ID":"1801c51a-c2ac-441f-a8f0-316c35aec026","Type":"ContainerStarted","Data":"0fad1b8188aa6db993a9f42c8e70ab2b1082011c9148664074ca97828ccc9b58"} Nov 02 21:21:19.196558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.196540 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:19.197909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.197885 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" event={"ID":"ffd53e7b-339d-4433-900b-141ea31a8b8a","Type":"ContainerStarted","Data":"aa5d172c63cc352db3d3429e7b57a2b06363b37cf03f2ed6c487b0335a20a229"} Nov 02 21:21:19.204139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.204082 5859 scope.go:122] "RemoveContainer" containerID="97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5" Nov 02 21:21:19.205131 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:19.205109 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5\": container with ID starting with 97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5 not found: ID does not exist" containerID="97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5" Nov 02 21:21:19.205222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.205141 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5"} err="failed to get container status \"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5\": rpc error: code = NotFound desc = could not find container \"97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5\": container with ID starting with 97fe50898a066fe39997698c7cb6b54ecd9a3acbe9021a2463f91bf36fe1c5f5 not found: ID does not exist" Nov 02 21:21:19.221546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.221525 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:21:19.281733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.281710 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\" (UniqueName: \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\") pod \"62ef207b-2d41-4bde-80a9-d022ac11185f\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " Nov 02 21:21:19.282311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.281814 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\") pod \"62ef207b-2d41-4bde-80a9-d022ac11185f\" (UID: \"62ef207b-2d41-4bde-80a9-d022ac11185f\") " Nov 02 21:21:19.282390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.282325 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume" pod "62ef207b-2d41-4bde-80a9-d022ac11185f" (UID: "62ef207b-2d41-4bde-80a9-d022ac11185f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:19.282390 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.282367 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\") pod \"e59a39e2-2d69-403f-8cb6-923897a84ab4\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " Nov 02 21:21:19.282740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.282719 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ef207b-2d41-4bde-80a9-d022ac11185f-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:19.282840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.282815 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume" pod "e59a39e2-2d69-403f-8cb6-923897a84ab4" (UID: "e59a39e2-2d69-403f-8cb6-923897a84ab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:19.284241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.284218 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh" pod "62ef207b-2d41-4bde-80a9-d022ac11185f" (UID: "62ef207b-2d41-4bde-80a9-d022ac11185f"). InnerVolumeSpecName "kube-api-access-v2thh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:19.383730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.383706 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\" (UniqueName: \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\") pod \"e59a39e2-2d69-403f-8cb6-923897a84ab4\" (UID: \"e59a39e2-2d69-403f-8cb6-923897a84ab4\") " Nov 02 21:21:19.384025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.384010 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2thh\" (UniqueName: \"kubernetes.io/projected/62ef207b-2d41-4bde-80a9-d022ac11185f-kube-api-access-v2thh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:19.384025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.384026 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59a39e2-2d69-403f-8cb6-923897a84ab4-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:19.385523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.385503 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz" pod "e59a39e2-2d69-403f-8cb6-923897a84ab4" (UID: "e59a39e2-2d69-403f-8cb6-923897a84ab4"). InnerVolumeSpecName "kube-api-access-ms2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:19.484505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.484486 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2mz\" (UniqueName: \"kubernetes.io/projected/e59a39e2-2d69-403f-8cb6-923897a84ab4-kube-api-access-ms2mz\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:19.544664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.544620 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" podStartSLOduration=1.544607778 podStartE2EDuration="1.544607778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:19.542345865 +0000 UTC m=+719.090595667" watchObservedRunningTime="2026-11-02 21:21:19.544607778 +0000 UTC m=+719.092857559" Nov 02 21:21:19.645766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.645710 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:21:19.650491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:19.650470 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-54a47302-778578d98c-dcq26"] Nov 02 21:21:20.206177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.206139 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" Nov 02 21:21:20.206177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.206158 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6" event={"ID":"e59a39e2-2d69-403f-8cb6-923897a84ab4","Type":"ContainerDied","Data":"3eee57f6d2c080e6b150d55973b1ea496bfe809d32e903a8a00241dacb0c2246"} Nov 02 21:21:20.206633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.206220 5859 scope.go:122] "RemoveContainer" containerID="1645e302a162a50c3863f04edaeddfbbfa161c165c3213066e3b2d9cb4b7c252" Nov 02 21:21:20.208171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.208142 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" event={"ID":"ffd53e7b-339d-4433-900b-141ea31a8b8a","Type":"ContainerStarted","Data":"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f"} Nov 02 21:21:20.208286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.208196 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:20.359150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.359129 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:21:20.362274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.362253 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-54a47302-85b4dfb47-qrjb6"] Nov 02 21:21:20.468273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.468186 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" podStartSLOduration=2.468174911 podStartE2EDuration="2.468174911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:20.468060178 +0000 UTC m=+720.016309956" watchObservedRunningTime="2026-11-02 21:21:20.468174911 +0000 UTC m=+720.016424702" Nov 02 21:21:20.570062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.570040 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ef207b-2d41-4bde-80a9-d022ac11185f" path="/var/lib/kubelet/pods/62ef207b-2d41-4bde-80a9-d022ac11185f/volumes" Nov 02 21:21:20.570389 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:20.570370 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59a39e2-2d69-403f-8cb6-923897a84ab4" path="/var/lib/kubelet/pods/e59a39e2-2d69-403f-8cb6-923897a84ab4/volumes" Nov 02 21:21:22.323911 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.323884 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 21:21:22.324387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324371 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e59a39e2-2d69-403f-8cb6-923897a84ab4" containerName="payload-pre-processing" Nov 02 21:21:22.324442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324387 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59a39e2-2d69-403f-8cb6-923897a84ab4" containerName="payload-pre-processing" Nov 02 21:21:22.324477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324440 5859 cpu_manager.go:414] "containerMap: removing container" podUID="62ef207b-2d41-4bde-80a9-d022ac11185f" containerName="payload-processing" Nov 02 21:21:22.324477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324449 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ef207b-2d41-4bde-80a9-d022ac11185f" containerName="payload-processing" Nov 02 21:21:22.324562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324549 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ef207b-2d41-4bde-80a9-d022ac11185f" containerName="payload-processing" Nov 02 21:21:22.324601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.324568 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59a39e2-2d69-403f-8cb6-923897a84ab4" containerName="payload-pre-processing" Nov 02 21:21:22.329381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.329361 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.379455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.379428 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:21:22.379553 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.379525 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d6cb38-serving-cert\"" Nov 02 21:21:22.412917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.412892 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvmd\" (UniqueName: \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.413064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.412933 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.413145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.413075 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.413145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.413135 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.426131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.426114 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 21:21:22.426131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.426117 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:21:22.426131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.426137 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:21:22.428199 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.428180 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 21:21:22.513803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.513777 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.513915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.513842 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.513986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.513916 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvmd\" (UniqueName: \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.514040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.513998 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.514153 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:22.514135 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-serving-cert: secret "maas-api-e2e-worker-w4-d6cb38-serving-cert" not found Nov 02 21:21:22.514220 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:22.514210 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls podName:9fc7b595-5f5c-48cc-9297-e30e154c6053 nodeName:}" failed. No retries permitted until 2026-11-02 21:21:23.014193832 +0000 UTC m=+722.562443609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls") pod "maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" (UID: "9fc7b595-5f5c-48cc-9297-e30e154c6053") : secret "maas-api-e2e-worker-w4-d6cb38-serving-cert" not found Nov 02 21:21:22.514472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.514455 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.516042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.516027 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.569220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.569196 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvmd\" (UniqueName: \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:22.938163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.938131 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:21:22.942351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:22.942324 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.004692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.004501 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f765a-openshift-default-dockercfg-6vjh4\"" Nov 02 21:21:23.009573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.009540 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:21:23.019104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019074 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019111 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019214 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019278 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019320 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019392 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mlb\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019433 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019451 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019565 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019509 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.019631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.019619 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:23.021821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.021800 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:23.120771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120738 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.120771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120768 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120809 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120835 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120862 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120881 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mlb\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120932 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.120986 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121036 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121160 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121266 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121346 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121439 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.121582 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.121566 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.123134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.123106 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.123412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.123392 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.147224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.147205 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 21:21:23.174507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.174483 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mlb\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.174610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.174592 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.242445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.242359 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:23.256368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.256349 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:23.679300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.679271 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:21:23.679577 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.679347 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 21:21:23.681796 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:23.681766 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed64869_14d9_4a52_ac1e_d78c1f79d95c.slice/crio-6a847a5e27e7da42823c03c1dfa52f2ed845a2c2fdf903c9e35b2d20ec9554da WatchSource:0}: Error finding container 6a847a5e27e7da42823c03c1dfa52f2ed845a2c2fdf903c9e35b2d20ec9554da: Status 404 returned error can't find the container with id 6a847a5e27e7da42823c03c1dfa52f2ed845a2c2fdf903c9e35b2d20ec9554da Nov 02 21:21:23.685070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.685026 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:23.685375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.685336 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:23.685478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.685417 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:23.729190 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:23.728042 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7b595_5f5c_48cc_9297_e30e154c6053.slice/crio-85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c WatchSource:0}: Error finding container 85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c: Status 404 returned error can't find the container with id 85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c Nov 02 21:21:23.729632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.729614 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:21:23.762303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.762287 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:21:23.828416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828385 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828446 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828479 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828517 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828569 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828621 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828663 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828706 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828732 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\") pod \"11e499c9-1d67-4017-9066-be401f8e311a\" (UID: \"11e499c9-1d67-4017-9066-be401f8e311a\") " Nov 02 21:21:23.828963 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.828888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:23.829118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829099 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:23.829313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829287 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.829313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829313 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e499c9-1d67-4017-9066-be401f8e311a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.829479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829331 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:23.829479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829345 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:23.829479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.829426 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:23.831244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.831223 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:21:23.831244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.831234 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:23.831556 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.831533 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "kube-api-access-5mbs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:23.831645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.831617 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy" pod "11e499c9-1d67-4017-9066-be401f8e311a" (UID: "11e499c9-1d67-4017-9066-be401f8e311a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:21:23.930757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930722 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930757 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930772 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e499c9-1d67-4017-9066-be401f8e311a-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930786 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930802 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930817 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e499c9-1d67-4017-9066-be401f8e311a-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:23.930943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:23.930830 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mbs7\" (UniqueName: \"kubernetes.io/projected/11e499c9-1d67-4017-9066-be401f8e311a-kube-api-access-5mbs7\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:24.239439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.239400 5859 generic.go:356] "Generic (PLEG): container finished" podID="11e499c9-1d67-4017-9066-be401f8e311a" containerID="2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954" exitCode=0 Nov 02 21:21:24.239599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.239436 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerDied","Data":"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954"} Nov 02 21:21:24.239599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.239466 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" Nov 02 21:21:24.239599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.239519 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c" event={"ID":"11e499c9-1d67-4017-9066-be401f8e311a","Type":"ContainerDied","Data":"6151b24e63c548e2ced0a7b0f8ed387fba58fac035143a2be58adda8d339b492"} Nov 02 21:21:24.239599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.239541 5859 scope.go:122] "RemoveContainer" containerID="2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954" Nov 02 21:21:24.244550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.244528 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerStarted","Data":"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad"} Nov 02 21:21:24.244664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.244571 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerStarted","Data":"6a847a5e27e7da42823c03c1dfa52f2ed845a2c2fdf903c9e35b2d20ec9554da"} Nov 02 21:21:24.246159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.246128 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" event={"ID":"9fc7b595-5f5c-48cc-9297-e30e154c6053","Type":"ContainerStarted","Data":"85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c"} Nov 02 21:21:24.252062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.251894 5859 scope.go:122] "RemoveContainer" containerID="0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7" Nov 02 21:21:24.257123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.257101 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:24.261042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.261023 5859 scope.go:122] "RemoveContainer" containerID="2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954" Nov 02 21:21:24.261316 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:24.261294 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954\": container with ID starting with 2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954 not found: ID does not exist" containerID="2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954" Nov 02 21:21:24.261378 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.261326 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954"} err="failed to get container status \"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954\": rpc error: code = NotFound desc = could not find container \"2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954\": container with ID starting with 2ac665a9e67eb3bf06df41576a0aa52a9df46e84dd7ce23ff55527d963a45954 not found: ID does not exist" Nov 02 21:21:24.261378 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.261349 5859 scope.go:122] "RemoveContainer" containerID="0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7" Nov 02 21:21:24.261598 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:24.261578 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7\": container with ID starting with 0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7 not found: ID does not exist" containerID="0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7" Nov 02 21:21:24.261708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.261600 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7"} err="failed to get container status \"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7\": rpc error: code = NotFound desc = could not find container \"0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7\": container with ID starting with 0657b24d3cde56b090269435f0ba071f5e80443fb96b5cd079ed6956ec7794c7 not found: ID does not exist" Nov 02 21:21:24.262322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.262308 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:24.396384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.396344 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podStartSLOduration=2.396333695 podStartE2EDuration="2.396333695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:24.396223809 +0000 UTC m=+723.944473607" watchObservedRunningTime="2026-11-02 21:21:24.396333695 +0000 UTC m=+723.944583469" Nov 02 21:21:24.500712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.500642 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:21:24.504594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.504569 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f4fe932-openshift-default-7659c4599-r7n7c"] Nov 02 21:21:24.570481 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:24.570444 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e499c9-1d67-4017-9066-be401f8e311a" path="/var/lib/kubelet/pods/11e499c9-1d67-4017-9066-be401f8e311a/volumes" Nov 02 21:21:25.255489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:25.255453 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" event={"ID":"9fc7b595-5f5c-48cc-9297-e30e154c6053","Type":"ContainerStarted","Data":"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b"} Nov 02 21:21:25.255989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:25.255513 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:25.255989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:25.255587 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:25.256624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:25.256603 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:21:25.367542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:25.367499 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" podStartSLOduration=2.482015832 podStartE2EDuration="3.36748315s" totalImagesPullingTime="885.467318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:25.367116775 +0000 UTC m=+724.915366577" watchObservedRunningTime="2026-11-02 21:21:25.36748315 +0000 UTC m=+724.915732932" Nov 02 21:21:25.943385 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:25.943350 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:21:27.891544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.891512 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 21:21:27.892250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.891988 5859 cpu_manager.go:414] "containerMap: removing container" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:27.892250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.891999 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:27.892250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.892083 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:27.895371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.895353 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:27.946822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.946799 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0c9014-serving-cert\"" Nov 02 21:21:27.976205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.976177 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:27.976327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.976236 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fss5\" (UniqueName: \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:27.976380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.976342 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:27.976468 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.976452 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:27.995499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:27.995476 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 21:21:28.077745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.077721 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fss5\" (UniqueName: \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.077866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.077774 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.077866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.077848 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.077938 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:28.077924 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-serving-cert: secret "maas-api-e2e-worker-w0-0c9014-serving-cert" not found Nov 02 21:21:28.078004 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:28.077993 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls podName:e13a11b4-7cc5-40fc-8183-27323975ce1e nodeName:}" failed. No retries permitted until 2026-11-02 21:21:28.577968711 +0000 UTC m=+728.126218488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls") pod "maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" (UID: "e13a11b4-7cc5-40fc-8183-27323975ce1e") : secret "maas-api-e2e-worker-w0-0c9014-serving-cert" not found Nov 02 21:21:28.078051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.078019 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.078455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.078426 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.080183 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.080159 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.130990 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.130966 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fss5\" (UniqueName: \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.180747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.180675 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 21:21:28.180924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.180779 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 21:21:28.583238 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.583200 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.585550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.585524 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:28.808083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:28.808055 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:29.222140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:29.222068 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 21:21:29.227509 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:29.227478 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13a11b4_7cc5_40fc_8183_27323975ce1e.slice/crio-8c838df68fb25e7197acb4a208a0f30842ba9ab74f46e0669bbe5a11148434fe WatchSource:0}: Error finding container 8c838df68fb25e7197acb4a208a0f30842ba9ab74f46e0669bbe5a11148434fe: Status 404 returned error can't find the container with id 8c838df68fb25e7197acb4a208a0f30842ba9ab74f46e0669bbe5a11148434fe Nov 02 21:21:29.276353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:29.276328 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" event={"ID":"e13a11b4-7cc5-40fc-8183-27323975ce1e","Type":"ContainerStarted","Data":"8c838df68fb25e7197acb4a208a0f30842ba9ab74f46e0669bbe5a11148434fe"} Nov 02 21:21:30.283566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:30.283525 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" event={"ID":"e13a11b4-7cc5-40fc-8183-27323975ce1e","Type":"ContainerStarted","Data":"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a"} Nov 02 21:21:30.283983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:30.283581 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:30.396001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:30.395907 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" podStartSLOduration=2.521736737 podStartE2EDuration="3.395892776s" totalImagesPullingTime="874.156039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:30.393409448 +0000 UTC m=+729.941659231" watchObservedRunningTime="2026-11-02 21:21:30.395892776 +0000 UTC m=+729.944142555" Nov 02 21:21:31.217450 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:31.217422 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 21:21:31.217646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:31.217496 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 21:21:31.264705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:31.264679 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 21:21:36.295135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:36.295064 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 21:21:49.391476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.391443 5859 generic.go:356] "Generic (PLEG): container finished" podID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerID="9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b" exitCode=0 Nov 02 21:21:49.391874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.391514 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerDied","Data":"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b"} Nov 02 21:21:49.391874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.391867 5859 scope.go:122] "RemoveContainer" containerID="9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b" Nov 02 21:21:49.393070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.393047 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:49.393123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.393106 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:49.393158 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:49.393135 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:50.398844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:50.398816 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerStarted","Data":"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1"} Nov 02 21:21:51.155230 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:51.155199 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:51.156557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:51.156529 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:21:51.156695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:51.156591 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 21:21:52.154623 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:52.154593 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:21:52.155016 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:52.154647 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 21:21:53.111072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:53.111039 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:53.155383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:53.155352 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:21:53.155746 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:53.155406 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 21:21:54.154602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:54.154565 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:21:54.154800 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:54.154630 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 21:21:55.154805 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.154767 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 21:21:55.155285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.154828 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 21:21:55.187170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.187138 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 21:21:55.187931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.187910 5859 cpu_manager.go:414] "containerMap: removing container" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:55.188006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.187937 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:55.188165 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.188148 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e499c9-1d67-4017-9066-be401f8e311a" containerName="istio-proxy" Nov 02 21:21:55.193027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.193005 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.244810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.244777 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 21:21:55.261688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.261665 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5f765a-serving-cert\"" Nov 02 21:21:55.284780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.284757 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.284905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.284872 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj5m6\" (UniqueName: \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.284905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.284901 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.285070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.285051 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.385940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.385906 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.386144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.386034 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj5m6\" (UniqueName: \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.386144 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:55.386048 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-serving-cert: secret "maas-api-e2e-worker-w1-5f765a-serving-cert" not found Nov 02 21:21:55.386144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.386074 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.386144 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:55.386121 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls podName:ecc52e39-8e8e-4aae-913b-1a46b6159d62 nodeName:}" failed. No retries permitted until 2026-11-02 21:21:55.886101816 +0000 UTC m=+755.434351600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls") pod "maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" (UID: "ecc52e39-8e8e-4aae-913b-1a46b6159d62") : secret "maas-api-e2e-worker-w1-5f765a-serving-cert" not found Nov 02 21:21:55.386366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.386152 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.386918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.386894 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.388519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.388498 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.444741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.444696 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj5m6\" (UniqueName: \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.890618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.890590 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.892908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:55.892878 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:55.943304 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:21:55.943275 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:21:56.110164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:56.110142 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:56.159590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:56.159562 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:56.161504 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:56.161475 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 21:21:56.482134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:56.482070 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 21:21:56.484981 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:21:56.484924 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc52e39_8e8e_4aae_913b_1a46b6159d62.slice/crio-370ad2f3c2e6fffe37b5cb9c40b0888e1c6221b03b3b7387e72feaebd1d3b502 WatchSource:0}: Error finding container 370ad2f3c2e6fffe37b5cb9c40b0888e1c6221b03b3b7387e72feaebd1d3b502: Status 404 returned error can't find the container with id 370ad2f3c2e6fffe37b5cb9c40b0888e1c6221b03b3b7387e72feaebd1d3b502 Nov 02 21:21:57.457396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:57.457358 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" event={"ID":"ecc52e39-8e8e-4aae-913b-1a46b6159d62","Type":"ContainerStarted","Data":"982ac9e2e162e04d48286893e423cc8609c3aa076231f2b29b19daeef5ac2676"} Nov 02 21:21:57.457717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:57.457418 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" event={"ID":"ecc52e39-8e8e-4aae-913b-1a46b6159d62","Type":"ContainerStarted","Data":"370ad2f3c2e6fffe37b5cb9c40b0888e1c6221b03b3b7387e72feaebd1d3b502"} Nov 02 21:21:57.457717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:57.457468 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:21:57.572509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:21:57.572466 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" podStartSLOduration=1.71121881 podStartE2EDuration="2.572453119s" totalImagesPullingTime="861.234309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:57.571267667 +0000 UTC m=+757.119517448" watchObservedRunningTime="2026-11-02 21:21:57.572453119 +0000 UTC m=+757.120702899" Nov 02 21:22:03.467449 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:03.467407 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 21:22:17.580705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.580664 5859 generic.go:356] "Generic (PLEG): container finished" podID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerID="ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad" exitCode=0 Nov 02 21:22:17.581288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.580732 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerDied","Data":"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad"} Nov 02 21:22:17.581288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.581109 5859 scope.go:122] "RemoveContainer" containerID="ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad" Nov 02 21:22:17.582969 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.582894 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:17.583111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.583091 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:17.583225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:17.583198 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:18.587887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:18.587859 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerStarted","Data":"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad"} Nov 02 21:22:19.257382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:19.257352 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:22:19.257550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:19.257478 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:22:19.257550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:19.257527 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:22:20.257143 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:20.257116 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:22:20.257521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:20.257170 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:22:21.257146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:21.257114 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:22:21.257327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:21.257163 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:22:22.257285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:22.257258 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:22:22.257455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:22.257310 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:22:23.256918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:23.256880 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:22:23.257230 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:23.256933 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:22:24.260965 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:24.260923 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:22:24.261435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:24.261421 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:22:24.262005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:22:24.261986 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:22:25.943873 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:22:25.943841 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:22:55.943352 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:22:55.943317 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:23:05.726461 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.726389 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 21:23:05.730507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.730489 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.784703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.784677 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 21:23:05.832827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.832801 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"openshift-service-ca.crt\"" Nov 02 21:23:05.832827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.832820 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"kube-root-ca.crt\"" Nov 02 21:23:05.832982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.832861 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"simulator-w1-5f765a-kserve-self-signed-certs\"" Nov 02 21:23:05.834133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.834119 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"default-dockercfg-tpxs7\"" Nov 02 21:23:05.870016 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.869988 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.870170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.870028 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.870170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.870053 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.870284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.870175 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.870284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.870255 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.870391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.870330 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxzf\" (UniqueName: \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971267 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971237 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971267 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971267 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971285 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971334 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxzf\" (UniqueName: \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971435 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971467 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971701 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971621 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971707 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.971809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.971795 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.973830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.973805 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:05.974120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:05.974099 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:06.028442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.028388 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxzf\" (UniqueName: \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\") pod \"simulator-w1-5f765a-kserve-77d5c979d8-7gnpl\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:06.068613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.068589 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:06.444535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.444511 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 21:23:06.446536 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:23:06.446508 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f246ab_a5d2_46b0_a8fa_8ff822401222.slice/crio-6a1e166affebaaabdba30dfd8e0b884627cf1c97cf1c8f2437ca574d1b4dbf2d WatchSource:0}: Error finding container 6a1e166affebaaabdba30dfd8e0b884627cf1c97cf1c8f2437ca574d1b4dbf2d: Status 404 returned error can't find the container with id 6a1e166affebaaabdba30dfd8e0b884627cf1c97cf1c8f2437ca574d1b4dbf2d Nov 02 21:23:06.867389 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.867353 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" event={"ID":"57f246ab-a5d2-46b0-a8fa-8ff822401222","Type":"ContainerStarted","Data":"66baee28c2984a09d5ac53c6cacfadba9abd1c5f2b2719cabd31d57f2cff92f0"} Nov 02 21:23:06.867794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.867397 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" event={"ID":"57f246ab-a5d2-46b0-a8fa-8ff822401222","Type":"ContainerStarted","Data":"6a1e166affebaaabdba30dfd8e0b884627cf1c97cf1c8f2437ca574d1b4dbf2d"} Nov 02 21:23:06.983070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:06.983026 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podStartSLOduration=1.9830096400000001 podStartE2EDuration="1.98300964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:06.980962519 +0000 UTC m=+826.529212293" watchObservedRunningTime="2026-11-02 21:23:06.98300964 +0000 UTC m=+826.531259427" Nov 02 21:23:07.069621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:07.069591 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:16.069696 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:16.069663 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:16.081667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:16.081645 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:16.935012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:23:16.934988 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:23:25.942839 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:23:25.942803 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:23:55.945007 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:23:55.944970 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:24:14.332888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.332849 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 21:24:14.338688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.338666 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.387528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.387499 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 21:24:14.397394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397369 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.397517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397406 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.397580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397544 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.397641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397583 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8vk\" (UniqueName: \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.397695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397648 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.397752 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.397713 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.435926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.435902 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"premium-w1-5f765a-kserve-self-signed-certs\"" Nov 02 21:24:14.499173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499138 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499188 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499265 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499290 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8vk\" (UniqueName: \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499310 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499351 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499532 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499629 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.499798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.499716 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.501522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.501486 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.501703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.501689 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.556643 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.556621 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8vk\" (UniqueName: \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\") pod \"premium-w1-5f765a-kserve-6879f74fd5-xxqwd\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:14.650731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:14.650675 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:15.029187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:15.029165 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 21:24:15.031453 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:24:15.031429 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a4997b2_60cb_42ee_b216_e4a93622b784.slice/crio-1461f134e37d30534c56bd7abd352b1d1da129f2d386280b98ae7f4cf96bbe6c WatchSource:0}: Error finding container 1461f134e37d30534c56bd7abd352b1d1da129f2d386280b98ae7f4cf96bbe6c: Status 404 returned error can't find the container with id 1461f134e37d30534c56bd7abd352b1d1da129f2d386280b98ae7f4cf96bbe6c Nov 02 21:24:15.254402 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:15.254367 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" event={"ID":"9a4997b2-60cb-42ee-b216-e4a93622b784","Type":"ContainerStarted","Data":"e3f201ed489bed5cf1087ab95d945184f9554ba18b79fda3345a37540af16431"} Nov 02 21:24:15.254574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:15.254416 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" event={"ID":"9a4997b2-60cb-42ee-b216-e4a93622b784","Type":"ContainerStarted","Data":"1461f134e37d30534c56bd7abd352b1d1da129f2d386280b98ae7f4cf96bbe6c"} Nov 02 21:24:15.384405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:15.384354 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podStartSLOduration=1.384338734 podStartE2EDuration="1.384338734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:15.38353735 +0000 UTC m=+894.931787142" watchObservedRunningTime="2026-11-02 21:24:15.384338734 +0000 UTC m=+894.932588516" Nov 02 21:24:15.650829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:15.650763 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:16.034383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:16.034319 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:24:16.034531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:16.034489 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" containerID="cri-o://7273f8710f44ff549b91654b616779f7923a69fce3a6574bf541d95c5c33a77d" gracePeriod=30 Nov 02 21:24:16.084772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:16.084746 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:24:16.084933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:16.084915 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" containerID="cri-o://2097bbf2bcca441f8483d66004360056710f375c7ec2ed46e32935bd8affc747" gracePeriod=30 Nov 02 21:24:20.173072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.173043 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:24:20.176998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.176979 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.229689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.229663 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d6cb38-openshift-default-dockercfg-mbhbl\"" Nov 02 21:24:20.254239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254213 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254243 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254264 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254342 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254504 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254481 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppvvd\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254551 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254525 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254616 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254685 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254667 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.254788 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.254768 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.282922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.282899 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:24:20.355306 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355280 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppvvd\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355318 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355342 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355376 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355403 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355491 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355515 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355596 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355664 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355714 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355747 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355790 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.355897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.355825 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.356251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.356234 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.357794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.357770 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.357907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.357812 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.412051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.412032 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.412243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.412226 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppvvd\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.490435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.490386 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:20.972796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.972770 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:24:20.974013 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:24:20.973990 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ab1639_650d_41fa_8f7b_366327d5de34.slice/crio-715d3ebaf7db0d1069724adf2b705bdd929da877860baa042d322cbe3f36ef01 WatchSource:0}: Error finding container 715d3ebaf7db0d1069724adf2b705bdd929da877860baa042d322cbe3f36ef01: Status 404 returned error can't find the container with id 715d3ebaf7db0d1069724adf2b705bdd929da877860baa042d322cbe3f36ef01 Nov 02 21:24:20.976197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.976167 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:20.976288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.976224 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:20.976288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:20.976255 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:21.315940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.315887 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" event={"ID":"a5ab1639-650d-41fa-8f7b-366327d5de34","Type":"ContainerStarted","Data":"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f"} Nov 02 21:24:21.316563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.316030 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" event={"ID":"a5ab1639-650d-41fa-8f7b-366327d5de34","Type":"ContainerStarted","Data":"715d3ebaf7db0d1069724adf2b705bdd929da877860baa042d322cbe3f36ef01"} Nov 02 21:24:21.320718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.320623 5859 generic.go:356] "Generic (PLEG): container finished" podID="1386322d-1fb7-4444-988f-4e4942c2d262" containerID="2097bbf2bcca441f8483d66004360056710f375c7ec2ed46e32935bd8affc747" exitCode=0 Nov 02 21:24:21.320718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.320707 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" event={"ID":"1386322d-1fb7-4444-988f-4e4942c2d262","Type":"ContainerDied","Data":"2097bbf2bcca441f8483d66004360056710f375c7ec2ed46e32935bd8affc747"} Nov 02 21:24:21.322932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.322799 5859 generic.go:356] "Generic (PLEG): container finished" podID="12df31b9-59d7-4528-869a-7f179fe61e65" containerID="7273f8710f44ff549b91654b616779f7923a69fce3a6574bf541d95c5c33a77d" exitCode=0 Nov 02 21:24:21.322932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.322859 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" event={"ID":"12df31b9-59d7-4528-869a-7f179fe61e65","Type":"ContainerDied","Data":"7273f8710f44ff549b91654b616779f7923a69fce3a6574bf541d95c5c33a77d"} Nov 02 21:24:21.340975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.340929 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:24:21.392072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.392054 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:24:21.435255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.435215 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podStartSLOduration=1.435198925 podStartE2EDuration="1.435198925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:21.433373554 +0000 UTC m=+900.981623337" watchObservedRunningTime="2026-11-02 21:24:21.435198925 +0000 UTC m=+900.983448708" Nov 02 21:24:21.466411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466387 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466418 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.466523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466442 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466497 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.466523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466524 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466556 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466587 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.466741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466652 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466653 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466746 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466781 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466807 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466841 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466868 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466901 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.466940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466926 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.466978 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467024 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467083 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\") pod \"1386322d-1fb7-4444-988f-4e4942c2d262\" (UID: \"1386322d-1fb7-4444-988f-4e4942c2d262\") " Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467120 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467123 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\") pod \"12df31b9-59d7-4528-869a-7f179fe61e65\" (UID: \"12df31b9-59d7-4528-869a-7f179fe61e65\") " Nov 02 21:24:21.467361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467300 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.467849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467332 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:24:21.467849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467580 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.467849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467794 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467940 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1386322d-1fb7-4444-988f-4e4942c2d262-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.467988 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468003 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468018 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468070 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468076 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468124 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468144 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.468604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468160 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.469133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.468663 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:24:21.469219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.469175 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:24:21.469279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.469243 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:24:21.469409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.469381 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "kube-api-access-ljltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:24:21.469578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.469556 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "kube-api-access-qk4mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:24:21.469852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.469827 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:24:21.470645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.470620 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:24:21.470820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.470800 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy" pod "1386322d-1fb7-4444-988f-4e4942c2d262" (UID: "1386322d-1fb7-4444-988f-4e4942c2d262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.470981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.470942 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy" pod "12df31b9-59d7-4528-869a-7f179fe61e65" (UID: "12df31b9-59d7-4528-869a-7f179fe61e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:24:21.490668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.490650 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:21.569437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569412 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1386322d-1fb7-4444-988f-4e4942c2d262-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569436 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569447 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12df31b9-59d7-4528-869a-7f179fe61e65-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569456 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12df31b9-59d7-4528-869a-7f179fe61e65-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569465 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12df31b9-59d7-4528-869a-7f179fe61e65-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569473 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1386322d-1fb7-4444-988f-4e4942c2d262-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569482 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4mk\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-kube-api-access-qk4mk\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569491 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569499 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/1386322d-1fb7-4444-988f-4e4942c2d262-kube-api-access-ljltx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:21.569584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:21.569507 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12df31b9-59d7-4528-869a-7f179fe61e65-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:24:22.335006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.334972 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" event={"ID":"1386322d-1fb7-4444-988f-4e4942c2d262","Type":"ContainerDied","Data":"17ddcb69d9cf7c86e2fb6833db6c7e96bc0ad39cbd44904682078d7fa2b10340"} Nov 02 21:24:22.335509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.335041 5859 scope.go:122] "RemoveContainer" containerID="2097bbf2bcca441f8483d66004360056710f375c7ec2ed46e32935bd8affc747" Nov 02 21:24:22.335509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.334984 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4" Nov 02 21:24:22.336507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.336489 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" Nov 02 21:24:22.336574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.336541 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj" event={"ID":"12df31b9-59d7-4528-869a-7f179fe61e65","Type":"ContainerDied","Data":"69eff0dfb1d1e261208d8f03cd9e53f1a3461570b99f006b40c1637c886b015d"} Nov 02 21:24:22.345139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.345123 5859 scope.go:122] "RemoveContainer" containerID="7273f8710f44ff549b91654b616779f7923a69fce3a6574bf541d95c5c33a77d" Nov 02 21:24:22.492635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.492601 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:24:22.492768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.492650 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:24:22.501213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.501184 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:24:22.551624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.551585 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-bd2cj"] Nov 02 21:24:22.575331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.575305 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" path="/var/lib/kubelet/pods/12df31b9-59d7-4528-869a-7f179fe61e65/volumes" Nov 02 21:24:22.632913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.632828 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:24:22.634899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:22.634880 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gkvm4"] Nov 02 21:24:23.490678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:23.490649 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:24:23.491079 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:23.490700 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:24:24.491760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:24.491724 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:24:24.492149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:24.491792 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:24:24.570101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:24.570070 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" path="/var/lib/kubelet/pods/1386322d-1fb7-4444-988f-4e4942c2d262/volumes" Nov 02 21:24:24.650808 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:24.650773 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:24.663913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:24.663888 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:25.366068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:25.366043 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:24:25.491422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:25.491391 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:24:25.491598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:25.491448 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:24:25.953661 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:24:25.953629 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:24:26.491699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:26.491671 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:24:26.491871 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:26.491718 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:24:27.603741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:27.603713 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:27.604250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:27.603981 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:27.604853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:27.604825 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:24:35.594843 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.594773 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:24:35.595273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595260 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" Nov 02 21:24:35.595314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595274 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" Nov 02 21:24:35.595314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595307 5859 cpu_manager.go:414] "containerMap: removing container" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" Nov 02 21:24:35.595314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595312 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" Nov 02 21:24:35.595405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595390 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="12df31b9-59d7-4528-869a-7f179fe61e65" containerName="istio-proxy" Nov 02 21:24:35.595405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.595402 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1386322d-1fb7-4444-988f-4e4942c2d262" containerName="istio-proxy" Nov 02 21:24:35.600302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.600283 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.627911 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.627881 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.627930 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.627980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628082 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628109 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628241 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628307 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628276 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628328 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wm8\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.628380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.628368 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.654766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.654745 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:24:35.703047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.703026 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\"" Nov 02 21:24:35.729008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.728989 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729021 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729039 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729101 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729140 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729224 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729260 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729293 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wm8\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729401 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729504 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729451 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729513 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729623 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.729830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.729809 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.730340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.730323 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.731337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.731313 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.731416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.731366 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.785908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.785882 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.786058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.786042 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wm8\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:35.913085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:35.913033 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:36.235772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.235700 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:24:36.240782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.240763 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.294272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.294226 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6fa533-openshift-default-dockercfg-psqwg\"" Nov 02 21:24:36.296805 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.296778 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:24:36.337218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337197 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337224 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337250 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337389 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337426 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337450 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337469 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337583 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbg8\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.337635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.337623 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.384763 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.384743 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:24:36.386110 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:24:36.386089 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d6fc9_3b5a_43d2_849b_61c16d3d2cba.slice/crio-26e9ecdedc1ed7fdb1bd564859e8de3e3b77bd6054516bc7d85ef0d34aa0377c WatchSource:0}: Error finding container 26e9ecdedc1ed7fdb1bd564859e8de3e3b77bd6054516bc7d85ef0d34aa0377c: Status 404 returned error can't find the container with id 26e9ecdedc1ed7fdb1bd564859e8de3e3b77bd6054516bc7d85ef0d34aa0377c Nov 02 21:24:36.388335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.388307 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:36.388421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.388362 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:36.388421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.388389 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:36.417666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.417645 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" event={"ID":"077d6fc9-3b5a-43d2-849b-61c16d3d2cba","Type":"ContainerStarted","Data":"26e9ecdedc1ed7fdb1bd564859e8de3e3b77bd6054516bc7d85ef0d34aa0377c"} Nov 02 21:24:36.438882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.438859 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.438887 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.438914 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.438997 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439032 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439066 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439096 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439144 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbg8\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439187 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439271 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439307 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439377 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439577 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439532 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.439793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.439771 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.441276 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.441256 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.441501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.441485 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.498181 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.498107 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbg8\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.498316 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.498245 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.554552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.554529 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:36.980910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.980880 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:24:36.982984 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:24:36.982931 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7eab2d_d875_4f9f_9eed_958e19d4c4dd.slice/crio-1d056d248da57dac6badf36160cf03263e2a97a6c3c971e855db1d76f752051c WatchSource:0}: Error finding container 1d056d248da57dac6badf36160cf03263e2a97a6c3c971e855db1d76f752051c: Status 404 returned error can't find the container with id 1d056d248da57dac6badf36160cf03263e2a97a6c3c971e855db1d76f752051c Nov 02 21:24:36.985364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.985331 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:36.985454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.985411 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:36.985454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:36.985440 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:37.436447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.436409 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" event={"ID":"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd","Type":"ContainerStarted","Data":"a67da46892b3e388f3cae87420dfaca96876d787b88a46f04f9d61eefd9c8abb"} Nov 02 21:24:37.436654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.436463 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" event={"ID":"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd","Type":"ContainerStarted","Data":"1d056d248da57dac6badf36160cf03263e2a97a6c3c971e855db1d76f752051c"} Nov 02 21:24:37.438245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.438215 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" event={"ID":"077d6fc9-3b5a-43d2-849b-61c16d3d2cba","Type":"ContainerStarted","Data":"11389928f507aa59b288c46ad3466a23c97f0b138d8c58a8338dc8985d790c69"} Nov 02 21:24:37.552525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.552475 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podStartSLOduration=1.552459345 podStartE2EDuration="1.552459345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:37.552217848 +0000 UTC m=+917.100467653" watchObservedRunningTime="2026-11-02 21:24:37.552459345 +0000 UTC m=+917.100709174" Nov 02 21:24:37.554979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.554940 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:37.694058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.693940 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podStartSLOduration=2.693926921 podStartE2EDuration="2.693926921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:37.693451955 +0000 UTC m=+917.241701746" watchObservedRunningTime="2026-11-02 21:24:37.693926921 +0000 UTC m=+917.242176680" Nov 02 21:24:37.925975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.920291 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:37.930145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.930105 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:24:37.930283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:37.930171 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 21:24:38.567139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:38.567097 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:38.568108 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:38.568028 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:38.917396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:38.917305 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:24:38.917652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:38.917621 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 21:24:39.555382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:39.555345 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:39.555561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:39.555405 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:39.924974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:39.924392 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:24:39.924974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:39.924469 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 21:24:40.555317 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:40.555289 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:40.555775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:40.555513 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:40.926381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:40.926346 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:24:40.926971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:40.926922 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 21:24:41.555320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:41.555281 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:41.555538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:41.555339 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:41.919366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:41.919330 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 02 21:24:41.919643 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:41.919613 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 02 21:24:42.555275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:42.555243 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:42.555628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:42.555295 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:42.952589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:42.952561 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:42.953164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:42.953137 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:42.953916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:42.953900 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:24:43.555540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:43.555508 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:24:43.556029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:43.555559 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:24:44.559106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:44.559079 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:44.559475 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:44.559445 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:44.560148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:24:44.560127 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:24:55.944780 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:24:55.944731 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:25:25.950362 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:25:25.950302 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:25:30.522707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.522673 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:25:30.528017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.527997 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.569034 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.569005 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.569141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.569050 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.569192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.569158 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.569292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.569276 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ctkl\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.581098 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.581067 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:25:30.628188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.628163 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-da55a9e52d6f\"" Nov 02 21:25:30.628188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.628169 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 21:25:30.628326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.628172 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 21:25:30.628326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.628190 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 21:25:30.631205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.631187 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-884fm\"" Nov 02 21:25:30.631205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.631195 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 21:25:30.669755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.669735 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ctkl\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.669927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.669816 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.669927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.669853 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.670090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.669945 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.670709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.670686 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.670837 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.670821 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.672198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.672182 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.726603 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.726580 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ctkl\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\") pod \"payload-processing-external-model-dbf8ff5d8-lxr5k\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:30.841285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:30.841262 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:31.052597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.052536 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:25:31.057587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.057567 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.114586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.114525 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:25:31.175398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175374 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175406 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175430 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175450 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175588 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175642 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175691 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175719 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.175743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.175734 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwxw\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.224783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.224761 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:25:31.226882 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:25:31.226855 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c064de_118a_4374_972e_59074311b153.slice/crio-4bdd54880302964e802bc19356f20a883437b911a6f29e2eee3e03809f8c38e1 WatchSource:0}: Error finding container 4bdd54880302964e802bc19356f20a883437b911a6f29e2eee3e03809f8c38e1: Status 404 returned error can't find the container with id 4bdd54880302964e802bc19356f20a883437b911a6f29e2eee3e03809f8c38e1 Nov 02 21:25:31.276437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276480 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276514 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276546 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276571 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwxw\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276689 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276719 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276757 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276770 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.276848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.276787 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.277052 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.277031 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.277250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.277225 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.277320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.277258 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.277418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.277403 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.278672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.278656 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.279227 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.279205 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.333614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.333591 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.333799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.333780 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwxw\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-v7zfc\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.371328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.371282 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:31.776134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.776056 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" event={"ID":"28c064de-118a-4374-972e-59074311b153","Type":"ContainerStarted","Data":"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e"} Nov 02 21:25:31.776134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.776121 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" event={"ID":"28c064de-118a-4374-972e-59074311b153","Type":"ContainerStarted","Data":"4bdd54880302964e802bc19356f20a883437b911a6f29e2eee3e03809f8c38e1"} Nov 02 21:25:31.776497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.776165 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:25:31.793780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.793748 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:25:31.800917 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:25:31.800894 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6590a39_bab1_46c2_9033_23d0dfa9302a.slice/crio-d0097fdddb2529f2b2f15be4dae72cc6cbd0d8f19688a8de9f71f6586c01569e WatchSource:0}: Error finding container d0097fdddb2529f2b2f15be4dae72cc6cbd0d8f19688a8de9f71f6586c01569e: Status 404 returned error can't find the container with id d0097fdddb2529f2b2f15be4dae72cc6cbd0d8f19688a8de9f71f6586c01569e Nov 02 21:25:31.803447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.803411 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:31.803541 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.803489 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:31.803541 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.803527 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:31.896451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:31.896400 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" podStartSLOduration=1.896387839 podStartE2EDuration="1.896387839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:31.893909842 +0000 UTC m=+971.442159648" watchObservedRunningTime="2026-11-02 21:25:31.896387839 +0000 UTC m=+971.444637621" Nov 02 21:25:32.783009 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:32.782976 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" event={"ID":"b6590a39-bab1-46c2-9033-23d0dfa9302a","Type":"ContainerStarted","Data":"cb51d478ac0105362ae226ab1cdc6414682cedf79a2fc06c6611d220394ea49f"} Nov 02 21:25:32.783423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:32.783029 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" event={"ID":"b6590a39-bab1-46c2-9033-23d0dfa9302a","Type":"ContainerStarted","Data":"d0097fdddb2529f2b2f15be4dae72cc6cbd0d8f19688a8de9f71f6586c01569e"} Nov 02 21:25:32.901665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:32.901607 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podStartSLOduration=1.901593381 podStartE2EDuration="1.901593381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:32.900241855 +0000 UTC m=+972.448491635" watchObservedRunningTime="2026-11-02 21:25:32.901593381 +0000 UTC m=+972.449843159" Nov 02 21:25:33.372419 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:33.372388 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:33.373930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:33.373910 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:33.374064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:33.373981 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:34.371594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:34.371561 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:34.371993 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:34.371622 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:35.371821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:35.371789 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:35.372244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:35.371849 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:36.371903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:36.371873 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:36.372313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:36.371921 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:37.372436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:37.372399 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:37.372842 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:37.372449 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:38.372448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:38.372413 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 21:25:38.372994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:38.372474 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 21:25:40.373073 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:40.373021 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:25:40.373567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:40.373091 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:25:41.415118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:41.415070 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:25:41.415615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:41.415127 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:25:42.449128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.449096 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:25:42.449558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.449146 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:25:42.450110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.450087 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:42.450500 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.450482 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:42.451031 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.451014 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:25:42.786992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:42.786883 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:46.056414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.056383 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:25:46.060440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.060423 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.153416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.153387 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:25:46.175690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.175668 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.175792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.175703 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.175792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.175734 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwlr\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.175919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.175891 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.175982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.175945 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.176095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.176073 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.176142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.176109 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.176254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.176236 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.176354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.176276 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277445 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277500 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277567 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277591 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277682 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277720 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277783 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277819 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277838 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277866 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwlr\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277910 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.277998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.277986 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.278342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.278219 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.278342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.278284 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.280036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.280015 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.280252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.280233 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.334839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.334813 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.335069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.335049 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwlr\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-qkm7n\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.380923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.380902 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:46.803144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.803113 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:25:46.804338 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:25:46.804314 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98db4240_4786_41aa_8b51_eba1dec7b156.slice/crio-e226c764ce40a2c05441529e75a0586bc46b731ca3a21d2f977d080bfc46fd9f WatchSource:0}: Error finding container e226c764ce40a2c05441529e75a0586bc46b731ca3a21d2f977d080bfc46fd9f: Status 404 returned error can't find the container with id e226c764ce40a2c05441529e75a0586bc46b731ca3a21d2f977d080bfc46fd9f Nov 02 21:25:46.806459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.806429 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:46.806546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.806487 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:46.806546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.806513 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:46.901584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.901553 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" event={"ID":"98db4240-4786-41aa-8b51-eba1dec7b156","Type":"ContainerStarted","Data":"0cb6bad1233308cc6dcc92f5995dc6ea26e789ab43138efa4ae34073dc366291"} Nov 02 21:25:46.901695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:46.901616 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" event={"ID":"98db4240-4786-41aa-8b51-eba1dec7b156","Type":"ContainerStarted","Data":"e226c764ce40a2c05441529e75a0586bc46b731ca3a21d2f977d080bfc46fd9f"} Nov 02 21:25:47.018849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:47.018802 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podStartSLOduration=1.018788659 podStartE2EDuration="1.018788659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:47.017476636 +0000 UTC m=+986.565726436" watchObservedRunningTime="2026-11-02 21:25:47.018788659 +0000 UTC m=+986.567038430" Nov 02 21:25:47.381571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:47.381537 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:47.382703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:47.382659 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:47.382861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:47.382723 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:48.381446 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:48.381410 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:48.381610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:48.381458 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:49.386706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:49.386283 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:49.386706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:49.386342 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:50.381787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:50.381755 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:50.381962 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:50.381810 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:51.381251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:51.381217 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:51.381609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:51.381275 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:52.382181 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:52.382150 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:52.382541 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:52.382201 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:52.784542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:52.784459 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:53.381291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:53.381253 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:53.381443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:53.381320 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:54.381973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:54.381911 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:25:54.382312 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:54.381991 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 21:25:55.385244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:55.385214 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:55.385705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:55.385601 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:55.386242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:25:55.386222 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:25:55.943432 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:25:55.943395 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:26:01.185105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:01.185068 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:26:01.186726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:01.186696 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:26:06.185908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.185806 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" containerID="cri-o://3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e" gracePeriod=25 Nov 02 21:26:06.458893 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.458869 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:26:06.518977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.518938 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\") pod \"28c064de-118a-4374-972e-59074311b153\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " Nov 02 21:26:06.519158 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519011 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\") pod \"28c064de-118a-4374-972e-59074311b153\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " Nov 02 21:26:06.519229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519165 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\") pod \"28c064de-118a-4374-972e-59074311b153\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " Nov 02 21:26:06.519229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519204 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\") pod \"28c064de-118a-4374-972e-59074311b153\" (UID: \"28c064de-118a-4374-972e-59074311b153\") " Nov 02 21:26:06.519336 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519314 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume" pod "28c064de-118a-4374-972e-59074311b153" (UID: "28c064de-118a-4374-972e-59074311b153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:26:06.519632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519599 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume" pod "28c064de-118a-4374-972e-59074311b153" (UID: "28c064de-118a-4374-972e-59074311b153"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:26:06.519770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519748 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:26:06.519834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.519768 5859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28c064de-118a-4374-972e-59074311b153-routing-overlay-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:26:06.521413 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.521390 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl" pod "28c064de-118a-4374-972e-59074311b153" (UID: "28c064de-118a-4374-972e-59074311b153"). InnerVolumeSpecName "kube-api-access-2ctkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:26:06.521512 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.521416 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials" pod "28c064de-118a-4374-972e-59074311b153" (UID: "28c064de-118a-4374-972e-59074311b153"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:26:06.622192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.622158 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ctkl\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-kube-api-access-2ctkl\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:26:06.622192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:06.622192 5859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28c064de-118a-4374-972e-59074311b153-provider-credentials\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:26:07.044723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.044461 5859 generic.go:356] "Generic (PLEG): container finished" podID="28c064de-118a-4374-972e-59074311b153" containerID="3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e" exitCode=0 Nov 02 21:26:07.044723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.044576 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" event={"ID":"28c064de-118a-4374-972e-59074311b153","Type":"ContainerDied","Data":"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e"} Nov 02 21:26:07.052988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.048492 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" event={"ID":"28c064de-118a-4374-972e-59074311b153","Type":"ContainerDied","Data":"4bdd54880302964e802bc19356f20a883437b911a6f29e2eee3e03809f8c38e1"} Nov 02 21:26:07.052988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.048622 5859 scope.go:122] "RemoveContainer" containerID="3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e" Nov 02 21:26:07.052988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.048873 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k" Nov 02 21:26:07.089642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.089618 5859 scope.go:122] "RemoveContainer" containerID="3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e" Nov 02 21:26:07.089939 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:26:07.089917 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e\": container with ID starting with 3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e not found: ID does not exist" containerID="3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e" Nov 02 21:26:07.090030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.089964 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e"} err="failed to get container status \"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e\": rpc error: code = NotFound desc = could not find container \"3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e\": container with ID starting with 3abe5e035257dadb33a50d7c3a52c6989246c07def945a6c1e95cd38e2d9c60e not found: ID does not exist" Nov 02 21:26:07.225017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.222911 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:26:07.272226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:07.272192 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-dbf8ff5d8-lxr5k"] Nov 02 21:26:08.591379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:08.591345 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c064de-118a-4374-972e-59074311b153" path="/var/lib/kubelet/pods/28c064de-118a-4374-972e-59074311b153/volumes" Nov 02 21:26:13.501074 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.501038 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:13.501456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.501109 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:13.877547 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.877506 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 21:26:13.878170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.878149 5859 cpu_manager.go:414] "containerMap: removing container" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" Nov 02 21:26:13.878170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.878170 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" Nov 02 21:26:13.878369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.878330 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c064de-118a-4374-972e-59074311b153" containerName="payload-processing" Nov 02 21:26:13.883359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.883341 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.889946 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.889918 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.890102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.890085 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.890156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.890118 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.890221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.890203 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.890311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.890291 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.890422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.890407 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsr7g\" (UniqueName: \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.964537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.964506 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 21:26:13.991522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.991482 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.991774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.991745 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.992021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.992003 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsr7g\" (UniqueName: \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.992193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.992176 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.992403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.992387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.992542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.992526 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.993544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.993527 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.994344 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.994325 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.994785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.994768 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:13.995489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:13.995458 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:14.011730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.011711 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"openshift-service-ca.crt\"" Nov 02 21:26:14.011835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.011756 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"default-dockercfg-lt567\"" Nov 02 21:26:14.011892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.011849 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"distinct-w2-6fa533-kserve-self-signed-certs\"" Nov 02 21:26:14.011981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.011939 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"kube-root-ca.crt\"" Nov 02 21:26:14.018877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.018644 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:14.051276 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.051251 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsr7g\" (UniqueName: \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\") pod \"distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:14.196043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.195968 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:14.571038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:14.571018 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 21:26:14.572232 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:26:14.572205 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66ed1713_c414_44f6_9370_0fc4a301881a.slice/crio-94fa6431494fba9dc5fb5a2f10be85397c2f7679366e58653431d9449c2c6702 WatchSource:0}: Error finding container 94fa6431494fba9dc5fb5a2f10be85397c2f7679366e58653431d9449c2c6702: Status 404 returned error can't find the container with id 94fa6431494fba9dc5fb5a2f10be85397c2f7679366e58653431d9449c2c6702 Nov 02 21:26:15.115848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:15.115817 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" event={"ID":"66ed1713-c414-44f6-9370-0fc4a301881a","Type":"ContainerStarted","Data":"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c"} Nov 02 21:26:15.116015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:15.115854 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" event={"ID":"66ed1713-c414-44f6-9370-0fc4a301881a","Type":"ContainerStarted","Data":"94fa6431494fba9dc5fb5a2f10be85397c2f7679366e58653431d9449c2c6702"} Nov 02 21:26:15.196581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:15.196556 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:15.230885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:15.230845 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podStartSLOduration=2.23083386 podStartE2EDuration="2.23083386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:15.23016139 +0000 UTC m=+1014.778411190" watchObservedRunningTime="2026-11-02 21:26:15.23083386 +0000 UTC m=+1014.779083640" Nov 02 21:26:20.863616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.863592 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:26:20.868983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.868942 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:20.922370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.922349 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 21:26:20.922474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.922371 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 21:26:20.922474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.922468 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 21:26:20.961893 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.961861 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rjv\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:20.962077 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.961906 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:20.962147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.962111 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:20.962254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.962233 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:20.972839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.972820 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 21:26:20.973052 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.973027 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-941bbc45395a\"" Nov 02 21:26:20.973159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.973063 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-532db23d7031\"" Nov 02 21:26:20.973159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.973071 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2ccbd1d6ff83\"" Nov 02 21:26:20.973159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.973115 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-884fm\"" Nov 02 21:26:20.975553 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:20.975536 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:26:21.063147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063092 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.063243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063152 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.063243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063202 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rjv\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.063243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063227 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.063766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063748 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.063897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.063879 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.065522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.065498 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.120365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.120341 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rjv\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\") pod \"payload-processing-external-model-57d7dc98b6-tgmpv\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.182141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.182117 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:21.554765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:21.554738 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:26:21.556483 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:26:21.556457 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416d4bd7_d4ef_4efc_a61d_14609d6aa05f.slice/crio-1c4a3c8f8b45df29e6f7ccdc0818f90707e9c270ce69e5d1bb5019718d59fc51 WatchSource:0}: Error finding container 1c4a3c8f8b45df29e6f7ccdc0818f90707e9c270ce69e5d1bb5019718d59fc51: Status 404 returned error can't find the container with id 1c4a3c8f8b45df29e6f7ccdc0818f90707e9c270ce69e5d1bb5019718d59fc51 Nov 02 21:26:22.162436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:22.162404 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" event={"ID":"416d4bd7-d4ef-4efc-a61d-14609d6aa05f","Type":"ContainerStarted","Data":"a0b2a09b0a465c287bba11920cfa2890fdfc26cdb050159b56a865c0c0b64f31"} Nov 02 21:26:22.162825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:22.162462 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" event={"ID":"416d4bd7-d4ef-4efc-a61d-14609d6aa05f","Type":"ContainerStarted","Data":"1c4a3c8f8b45df29e6f7ccdc0818f90707e9c270ce69e5d1bb5019718d59fc51"} Nov 02 21:26:22.162825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:22.162507 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:22.280112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:22.280060 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" podStartSLOduration=2.280046849 podStartE2EDuration="2.280046849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:22.277102444 +0000 UTC m=+1021.825352235" watchObservedRunningTime="2026-11-02 21:26:22.280046849 +0000 UTC m=+1021.828296629" Nov 02 21:26:24.196853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:24.196825 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:24.209663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:24.209633 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:25.191921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:25.191890 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:26:25.960654 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:26:25.960606 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:26:28.497125 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:28.497085 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-v7zfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:28.497545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:28.497162 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:33.171219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:33.171189 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:26:52.226087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.226054 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:26:52.230463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.230436 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.344740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.344713 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.344885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.344794 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.344941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.344911 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345126 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345179 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345263 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345291 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345312 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2845j\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.345494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.345475 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.423342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.423316 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:26:52.447113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447091 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447144 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447182 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447230 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447254 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447280 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2845j\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447382 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447461 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447425 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447530 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447515 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447653 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447753 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447708 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447871 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447849 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.447938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.447894 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.448129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.448102 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.449760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.449729 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.449851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.449825 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.503710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.503655 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.503710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.503673 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2845j\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.546928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.546897 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:52.972387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.972365 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:26:52.973894 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:26:52.973869 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ace614_33ba_4f58_b987_25e50dd83dd8.slice/crio-eeeba8b0bf20dc48aa8ad3eea1ea43a435277c2375a907c5408a11b9701dea5b WatchSource:0}: Error finding container eeeba8b0bf20dc48aa8ad3eea1ea43a435277c2375a907c5408a11b9701dea5b: Status 404 returned error can't find the container with id eeeba8b0bf20dc48aa8ad3eea1ea43a435277c2375a907c5408a11b9701dea5b Nov 02 21:26:52.976021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.975992 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:52.976134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.976066 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:52.976134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:52.976108 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:53.389509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:53.389472 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" event={"ID":"c4ace614-33ba-4f58-b987-25e50dd83dd8","Type":"ContainerStarted","Data":"587d85bc4f4297f7fef0a229dd48273cce4b5201392d2751060a9fb4271a12bd"} Nov 02 21:26:53.390024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:53.389522 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" event={"ID":"c4ace614-33ba-4f58-b987-25e50dd83dd8","Type":"ContainerStarted","Data":"eeeba8b0bf20dc48aa8ad3eea1ea43a435277c2375a907c5408a11b9701dea5b"} Nov 02 21:26:53.547859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:53.547829 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:54.549580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:54.549546 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:26:54.549942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:54.549596 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:26:55.547868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:55.547834 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:26:55.548061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:55.547886 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:26:55.957915 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:26:55.957881 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:26:56.547924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:56.547887 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:26:56.548122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:56.547944 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:26:57.547733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:57.547700 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:26:57.548216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:57.547752 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:26:58.548001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:58.547937 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:26:58.548376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:58.548014 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:26:59.663379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:59.663336 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:59.663811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:59.663647 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:59.664469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:59.664451 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:26:59.824739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:26:59.824694 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podStartSLOduration=7.824683169 podStartE2EDuration="7.824683169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:53.511271085 +0000 UTC m=+1053.059520892" watchObservedRunningTime="2026-11-02 21:26:59.824683169 +0000 UTC m=+1059.372933008" Nov 02 21:27:06.360102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.360066 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:27:06.364293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.364277 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.461401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.461374 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:27:06.515042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515017 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515056 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515193 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515223 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515260 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntzr\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515361 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515390 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515467 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515425 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.515517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.515500 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.616817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616744 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.616817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616783 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.616817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616815 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntzr\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616887 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616926 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.616984 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617021 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617139 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617172 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617196 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617253 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617518 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617301 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617518 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617493 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.617848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.617827 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.619344 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.619321 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.619457 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.619357 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.673454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.673434 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.673726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.673710 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntzr\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:06.678078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:06.678060 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:07.101499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.101477 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:27:07.103259 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:27:07.103234 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2a5e75_9891_41f5_916a_a837f2f73f4d.slice/crio-d3591367c453f5c45ab18511aa10764ab830242f16ba0df3f28670df9c6e514d WatchSource:0}: Error finding container d3591367c453f5c45ab18511aa10764ab830242f16ba0df3f28670df9c6e514d: Status 404 returned error can't find the container with id d3591367c453f5c45ab18511aa10764ab830242f16ba0df3f28670df9c6e514d Nov 02 21:27:07.105627 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.105599 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:27:07.105700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.105657 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:27:07.105700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.105687 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:27:07.497417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.497338 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" event={"ID":"3e2a5e75-9891-41f5-916a-a837f2f73f4d","Type":"ContainerStarted","Data":"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977"} Nov 02 21:27:07.497417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.497400 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" event={"ID":"3e2a5e75-9891-41f5-916a-a837f2f73f4d","Type":"ContainerStarted","Data":"d3591367c453f5c45ab18511aa10764ab830242f16ba0df3f28670df9c6e514d"} Nov 02 21:27:07.678490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:07.678453 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:08.679786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:08.679755 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:27:08.680222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:08.679814 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:27:09.678874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:09.678843 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:27:09.679067 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:09.678914 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:27:10.678895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:10.678860 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:27:10.679278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:10.678911 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:27:11.679228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:11.679192 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:27:11.679605 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:11.679242 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:27:12.678930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:12.678897 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:27:12.679119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:12.678947 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:27:14.046036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:14.045298 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:14.046576 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:14.046075 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:14.047057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:14.047038 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:27:14.213598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:27:14.213546 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podStartSLOduration=8.21353432 podStartE2EDuration="8.21353432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:07.628621367 +0000 UTC m=+1067.176871150" watchObservedRunningTime="2026-11-02 21:27:14.21353432 +0000 UTC m=+1073.761784100" Nov 02 21:27:25.943584 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:27:25.943551 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:27:55.944649 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:27:55.944610 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:28:25.943203 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:28:25.943170 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:28:45.003163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.003132 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 21:28:45.007260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.007240 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.062225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.062195 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 21:28:45.109117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.109094 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"openshift-service-ca.crt\"" Nov 02 21:28:45.109269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.109255 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"kube-root-ca.crt\"" Nov 02 21:28:45.110437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.110424 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"distinct-w4-d6cb38-kserve-self-signed-certs\"" Nov 02 21:28:45.110538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.110528 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"default-dockercfg-rtxrz\"" Nov 02 21:28:45.122771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.122746 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trmb\" (UniqueName: \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.122870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.122788 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.122909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.122888 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.123031 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.123018 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.123071 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.123037 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.123071 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.123062 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224008 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224039 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224036 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224067 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224113 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trmb\" (UniqueName: \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224146 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224209 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224452 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224473 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.224620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.224580 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.226438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.226416 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.226615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.226597 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.280819 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.280764 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trmb\" (UniqueName: \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\") pod \"distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.319449 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.319428 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:45.694123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:45.694099 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 21:28:45.695959 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:28:45.695924 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7a1c035_2d96_4e95_8da5_a0642d874fb5.slice/crio-e9e74f3aae79bf5c7fd71b0ff3eefd61f04c33c5140e6f86bd672246c394ec30 WatchSource:0}: Error finding container e9e74f3aae79bf5c7fd71b0ff3eefd61f04c33c5140e6f86bd672246c394ec30: Status 404 returned error can't find the container with id e9e74f3aae79bf5c7fd71b0ff3eefd61f04c33c5140e6f86bd672246c394ec30 Nov 02 21:28:46.130827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:46.130797 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" event={"ID":"b7a1c035-2d96-4e95-8da5-a0642d874fb5","Type":"ContainerStarted","Data":"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1"} Nov 02 21:28:46.131233 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:46.130836 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" event={"ID":"b7a1c035-2d96-4e95-8da5-a0642d874fb5","Type":"ContainerStarted","Data":"e9e74f3aae79bf5c7fd71b0ff3eefd61f04c33c5140e6f86bd672246c394ec30"} Nov 02 21:28:46.245917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:46.245873 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podStartSLOduration=2.245860133 podStartE2EDuration="2.245860133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:46.245339581 +0000 UTC m=+1165.793589363" watchObservedRunningTime="2026-11-02 21:28:46.245860133 +0000 UTC m=+1165.794109913" Nov 02 21:28:46.320130 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:46.320091 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:55.319798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:55.319767 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:55.331350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:55.331328 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:28:55.963995 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:28:55.963911 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:28:56.227500 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:28:56.227417 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:29:08.174335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.174256 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:29:08.178832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.178808 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.224894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.224867 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:29:08.228666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.228647 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.234361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.234339 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:29:08.292706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.292677 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:29:08.359349 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359322 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359368 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359384 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359427 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359497 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmgh\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359574 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359593 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359612 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359668 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kz5\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359776 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359834 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359863 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359914 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.359991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359960 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.359991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.359986 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.360122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.360006 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.360122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.360044 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.360188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.360135 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461066 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461105 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461122 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461144 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461185 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461213 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461243 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461275 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461307 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461354 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461403 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461427 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461453 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461482 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmgh\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461615 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461644 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461674 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.461710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461691 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.462162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461732 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.462162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461752 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kz5\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.462162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.461907 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.462162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462121 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.462374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462243 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.462427 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462413 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.462478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462449 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.462536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462509 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.462616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462593 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.462744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.462721 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.464160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.464139 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.464282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.464239 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.464450 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.464430 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.464726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.464708 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.518701 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.518671 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.519082 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.519060 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmgh\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.519423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.519406 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.519488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.519413 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kz5\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.541388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.541367 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:08.793326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.793247 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:08.971883 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.971858 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:29:08.973130 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:29:08.973103 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dd1bfb_1d46_49bb_a209_84f046e20f9c.slice/crio-bd4f8010438b7528956614815a30471713c8368902daa851626ae9fe7db8568c WatchSource:0}: Error finding container bd4f8010438b7528956614815a30471713c8368902daa851626ae9fe7db8568c: Status 404 returned error can't find the container with id bd4f8010438b7528956614815a30471713c8368902daa851626ae9fe7db8568c Nov 02 21:29:08.975363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.975336 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:08.975442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.975398 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:08.975442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:08.975427 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:09.171099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.171070 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:29:09.171646 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:29:09.171619 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669ae170_5bfa_4bbe_8037_5d510aca4f7f.slice/crio-53e2a503aaec9aaa52397a0a161931a407091e13ece5f85b1a28e0d4bceb913d WatchSource:0}: Error finding container 53e2a503aaec9aaa52397a0a161931a407091e13ece5f85b1a28e0d4bceb913d: Status 404 returned error can't find the container with id 53e2a503aaec9aaa52397a0a161931a407091e13ece5f85b1a28e0d4bceb913d Nov 02 21:29:09.174058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.174026 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:09.174151 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.174100 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:09.174206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.174145 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:09.305494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.305398 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" event={"ID":"c3dd1bfb-1d46-49bb-a209-84f046e20f9c","Type":"ContainerStarted","Data":"3219815b3d66acefddc2dd296c9fd983636a6274310c4bd318c344a40b163727"} Nov 02 21:29:09.305494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.305460 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" event={"ID":"c3dd1bfb-1d46-49bb-a209-84f046e20f9c","Type":"ContainerStarted","Data":"bd4f8010438b7528956614815a30471713c8368902daa851626ae9fe7db8568c"} Nov 02 21:29:09.307584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.307558 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" event={"ID":"669ae170-5bfa-4bbe-8037-5d510aca4f7f","Type":"ContainerStarted","Data":"7fce3676056670c607412de9fe46a5dd6e0d9b5061991b85f063a58f7afc39f0"} Nov 02 21:29:09.307713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.307617 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" event={"ID":"669ae170-5bfa-4bbe-8037-5d510aca4f7f","Type":"ContainerStarted","Data":"53e2a503aaec9aaa52397a0a161931a407091e13ece5f85b1a28e0d4bceb913d"} Nov 02 21:29:09.436246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.436196 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podStartSLOduration=1.436182708 podStartE2EDuration="1.436182708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:09.435367507 +0000 UTC m=+1188.983617309" watchObservedRunningTime="2026-11-02 21:29:09.436182708 +0000 UTC m=+1188.984432488" Nov 02 21:29:09.542025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.541990 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:09.556329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.556136 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podStartSLOduration=1.556122052 podStartE2EDuration="1.556122052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:09.552770608 +0000 UTC m=+1189.101020407" watchObservedRunningTime="2026-11-02 21:29:09.556122052 +0000 UTC m=+1189.104371832" Nov 02 21:29:09.794356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:09.794323 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:10.552976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:10.552051 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:10.552976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:10.552124 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:10.794902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:10.794869 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:10.795116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:10.794934 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:11.552978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:11.542375 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:11.552978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:11.542441 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:11.793729 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:11.793696 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:11.793912 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:11.793748 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:12.547975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:12.547400 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:12.547975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:12.547457 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:12.794478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:12.794444 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:12.794857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:12.794494 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:13.542768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:13.542355 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:13.542768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:13.542413 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:13.794498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:13.794406 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:13.794498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:13.794462 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:14.543397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:14.543347 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:14.543693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:14.543657 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:14.794229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:14.794149 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:14.794229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:14.794203 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:15.542882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:15.542835 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:15.543487 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:15.542900 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 21:29:15.793902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:15.793795 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:29:15.793902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:15.793872 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:29:17.588202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:17.588133 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:29:17.589228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:17.588875 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:17.798002 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:17.797934 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:29:17.798172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:17.798052 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:18.335187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.335154 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:18.336066 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.336038 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:18.336877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.336841 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:29:18.570112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.570085 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:18.570364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.570348 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:18.571071 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:18.571051 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:29:25.943285 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:29:25.943242 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:29:26.502545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.502512 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:29:26.507899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.507879 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.600780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.600753 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.600809 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.600987 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601026 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601121 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601152 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601275 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601309 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.601548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.601422 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9gf\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.604785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.604766 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:29:26.702208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702185 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702213 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702248 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9gf\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702309 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702352 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702396 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702420 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702479 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702704 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702512 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.702860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702833 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.703003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.702900 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.703096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.703006 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.703096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.703048 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.703212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.703148 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.704839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.704816 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.704919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.704899 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.758934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.758880 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.759097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.759077 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9gf\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:26.821754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:26.821729 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:27.246001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.245978 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:29:27.246768 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:29:27.246743 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f973eaa_27f9_4197_8c9d_1e6b0aafa86f.slice/crio-8939db874d6065554b97251ab00f969fad51ad0f41eac43eabc9aa67aea57818 WatchSource:0}: Error finding container 8939db874d6065554b97251ab00f969fad51ad0f41eac43eabc9aa67aea57818: Status 404 returned error can't find the container with id 8939db874d6065554b97251ab00f969fad51ad0f41eac43eabc9aa67aea57818 Nov 02 21:29:27.249114 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.249081 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:27.249185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.249147 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:27.249219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.249185 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:27.544060 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.543963 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" event={"ID":"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f","Type":"ContainerStarted","Data":"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6"} Nov 02 21:29:27.544060 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.544017 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" event={"ID":"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f","Type":"ContainerStarted","Data":"8939db874d6065554b97251ab00f969fad51ad0f41eac43eabc9aa67aea57818"} Nov 02 21:29:27.685381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.685327 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podStartSLOduration=1.685313536 podStartE2EDuration="1.685313536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:27.682271666 +0000 UTC m=+1207.230521447" watchObservedRunningTime="2026-11-02 21:29:27.685313536 +0000 UTC m=+1207.233563316" Nov 02 21:29:27.822168 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:27.822149 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:28.831618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:28.831578 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:28.832054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:28.831630 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:29.823415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:29.823368 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:29.823728 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:29.823695 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:30.825975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:30.825283 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:30.825975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:30.825354 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:31.825975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:31.825254 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:31.825975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:31.825315 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:32.822374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:32.822335 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:32.822654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:32.822627 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:33.186335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:33.186241 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:33.205435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:33.205396 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:33.826357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:33.826321 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:33.826655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:33.826617 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:29:35.863115 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:35.863068 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:29:35.863524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:35.863130 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:36.161161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:36.161082 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:36.161639 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:36.161614 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:36.162774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:36.162756 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:29:41.636782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.636747 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:29:41.640943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.640926 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.687130 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.687109 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:29:41.690937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.690917 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.739850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.739818 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:29:41.772664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.772634 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:29:41.821363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821334 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.821508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821373 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtknl\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821393 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821462 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821612 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.821682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821669 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821698 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821726 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821779 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.821838 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821803 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.821879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821847 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.821921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821875 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.821921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.821909 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.822012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.822000 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.822044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.822022 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.822086 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.822041 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.822119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.822096 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7jw\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.822152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.822126 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923264 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923297 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923318 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923345 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923376 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923415 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923441 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923474 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923506 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923534 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923601 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7jw\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923645 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923745 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923748 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923777 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923786 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtknl\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923838 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.923977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923891 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.924424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.923993 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.924424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924003 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.924424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924094 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.924424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924283 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.924625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924464 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.924625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924543 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.924740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924694 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.924740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924717 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.925018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.924974 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.926711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.926687 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.926836 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.926791 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.926905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.926864 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.927053 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.927035 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.980921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.980896 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.981047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.980899 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtknl\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:41.981090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.981046 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7jw\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:41.981335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:41.981319 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:42.006409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.006389 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:42.256099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.256027 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:42.435835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.435810 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:29:42.436760 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:29:42.436735 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7276909f_cbc2_42a5_bc13_c844cfcd31d8.slice/crio-0f33c7cf6703c23c335bdefe0dbdb9e8fe22a388e37649e192915714eacb62b0 WatchSource:0}: Error finding container 0f33c7cf6703c23c335bdefe0dbdb9e8fe22a388e37649e192915714eacb62b0: Status 404 returned error can't find the container with id 0f33c7cf6703c23c335bdefe0dbdb9e8fe22a388e37649e192915714eacb62b0 Nov 02 21:29:42.438888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.438859 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.438965 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.438919 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.439010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.438961 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.636318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.636290 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:29:42.637062 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:29:42.637038 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012bc30b_1291_4f15_be9a_fcfcd851491f.slice/crio-68ef6a61b06c76b6140ef4653ac1f71edc9e96f1565d124ebe1f9403427a23db WatchSource:0}: Error finding container 68ef6a61b06c76b6140ef4653ac1f71edc9e96f1565d124ebe1f9403427a23db: Status 404 returned error can't find the container with id 68ef6a61b06c76b6140ef4653ac1f71edc9e96f1565d124ebe1f9403427a23db Nov 02 21:29:42.639640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.639606 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.639709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.639694 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.639749 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.639735 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.726795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.726762 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" event={"ID":"7276909f-cbc2-42a5-bc13-c844cfcd31d8","Type":"ContainerStarted","Data":"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb"} Nov 02 21:29:42.727021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.726831 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" event={"ID":"7276909f-cbc2-42a5-bc13-c844cfcd31d8","Type":"ContainerStarted","Data":"0f33c7cf6703c23c335bdefe0dbdb9e8fe22a388e37649e192915714eacb62b0"} Nov 02 21:29:42.728770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.728750 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" event={"ID":"012bc30b-1291-4f15-be9a-fcfcd851491f","Type":"ContainerStarted","Data":"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624"} Nov 02 21:29:42.728886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.728791 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" event={"ID":"012bc30b-1291-4f15-be9a-fcfcd851491f","Type":"ContainerStarted","Data":"68ef6a61b06c76b6140ef4653ac1f71edc9e96f1565d124ebe1f9403427a23db"} Nov 02 21:29:42.845771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.845710 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podStartSLOduration=1.84569374 podStartE2EDuration="1.84569374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:42.843407973 +0000 UTC m=+1222.391657753" watchObservedRunningTime="2026-11-02 21:29:42.84569374 +0000 UTC m=+1222.393943522" Nov 02 21:29:42.965201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:42.965089 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podStartSLOduration=1.965071239 podStartE2EDuration="1.965071239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:42.962375661 +0000 UTC m=+1222.510625455" watchObservedRunningTime="2026-11-02 21:29:42.965071239 +0000 UTC m=+1222.513321020" Nov 02 21:29:43.020638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.019297 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:43.031655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.031253 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:43.031655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.031310 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:43.266063 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.261127 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:43.266063 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.262677 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:43.266063 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:43.262739 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:44.022975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:44.021334 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:44.022975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:44.021393 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:44.257975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:44.256414 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:44.257975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:44.256476 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:45.018977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:45.017299 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:45.018977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:45.017360 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:45.263976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:45.263555 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:45.263976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:45.263630 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:46.013976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:46.009261 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:46.013976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:46.009315 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:46.257215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:46.257178 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:46.257385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:46.257238 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:47.020973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:47.009265 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.020973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:47.009323 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:47.257976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:47.257260 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.257976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:47.257323 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:48.012974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:48.009271 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.012974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:48.009328 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:48.258010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:48.257963 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.258557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:48.258054 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:49.007690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:49.007653 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:29:49.007942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:49.007698 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:29:49.256718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:49.256680 5859 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:29:49.256881 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:49.256746 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:29:50.011602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.011571 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:50.012084 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.011937 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:50.012608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.012590 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:29:50.260705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.260677 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:50.260980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.260940 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:50.261725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:50.261675 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:29:56.153990 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:29:56.151563 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:29:56.217976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.207019 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.217976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.207022 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.297809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.297771 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" podUID="8e0ac144-950e-424b-9461-6a2333d05969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.325975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.322111 5859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.364975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.364007 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.932925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:56.932016 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 21:29:57.233037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.231738 5859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 21:29:57.237041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.235997 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:57.237041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.236301 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:29:57.250796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.234018 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:57.251387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.251317 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:29:57.350978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.343642 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" podUID="8e0ac144-950e-424b-9461-6a2333d05969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:57.350978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.344083 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:29:57.374154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.372231 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:57.374154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.372583 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:29:57.761770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.761730 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 21:29:57.799433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.799407 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" Nov 02 21:29:57.865033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.865003 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:29:57.866903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:29:57.866881 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:30:00.172214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.172135 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:00.177650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.177629 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.230032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.230006 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 21:30:00.275903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.275872 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.276080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.275933 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xjn\" (UniqueName: \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.283356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.283331 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:00.377378 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.377347 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.377519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.377428 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xjn\" (UniqueName: \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.378038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.378019 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.434806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.434744 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xjn\" (UniqueName: \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.493856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.493823 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:00.874577 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.874534 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:00.880885 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:30:00.880855 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1633e410_736f_422a_b350_c6bddd09042a.slice/crio-79ce58bdf0c822fa407335c6d5ff53e4c97b37aa607eb33059a2a71132a027b8 WatchSource:0}: Error finding container 79ce58bdf0c822fa407335c6d5ff53e4c97b37aa607eb33059a2a71132a027b8: Status 404 returned error can't find the container with id 79ce58bdf0c822fa407335c6d5ff53e4c97b37aa607eb33059a2a71132a027b8 Nov 02 21:30:00.883386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.883369 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:30:00.944762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:00.944735 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerStarted","Data":"79ce58bdf0c822fa407335c6d5ff53e4c97b37aa607eb33059a2a71132a027b8"} Nov 02 21:30:05.979992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:05.979934 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerStarted","Data":"316c9d3d3f4523299939e681f72802ce17e90e11d50994b3a6a25575d31f1659"} Nov 02 21:30:06.098894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:06.098820 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" podStartSLOduration=1.460053523 podStartE2EDuration="6.098801687s" totalImagesPullingTime="4.638748164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:06.095163263 +0000 UTC m=+1245.643413068" watchObservedRunningTime="2026-11-02 21:30:06.098801687 +0000 UTC m=+1245.647051468" Nov 02 21:30:06.988195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:06.988158 5859 generic.go:356] "Generic (PLEG): container finished" podID="1633e410-736f-422a-b350-c6bddd09042a" containerID="316c9d3d3f4523299939e681f72802ce17e90e11d50994b3a6a25575d31f1659" exitCode=60 Nov 02 21:30:06.988571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:06.988253 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerDied","Data":"316c9d3d3f4523299939e681f72802ce17e90e11d50994b3a6a25575d31f1659"} Nov 02 21:30:06.988571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:06.988482 5859 scope.go:122] "RemoveContainer" containerID="316c9d3d3f4523299939e681f72802ce17e90e11d50994b3a6a25575d31f1659" Nov 02 21:30:07.996366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:07.996331 5859 generic.go:356] "Generic (PLEG): container finished" podID="1633e410-736f-422a-b350-c6bddd09042a" containerID="b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9" exitCode=60 Nov 02 21:30:07.996366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:07.996371 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerDied","Data":"b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9"} Nov 02 21:30:07.996867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:07.996419 5859 scope.go:122] "RemoveContainer" containerID="316c9d3d3f4523299939e681f72802ce17e90e11d50994b3a6a25575d31f1659" Nov 02 21:30:07.996867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:07.996728 5859 scope.go:122] "RemoveContainer" containerID="b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9" Nov 02 21:30:07.997078 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:07.997053 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc_odh-ai-gateway-infra(1633e410-736f-422a-b350-c6bddd09042a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" podUID="1633e410-736f-422a-b350-c6bddd09042a" Nov 02 21:30:09.005140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:09.005115 5859 scope.go:122] "RemoveContainer" containerID="b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9" Nov 02 21:30:09.005524 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:09.005375 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc_odh-ai-gateway-infra(1633e410-736f-422a-b350-c6bddd09042a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" podUID="1633e410-736f-422a-b350-c6bddd09042a" Nov 02 21:30:12.372371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.372342 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:12.376703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.376681 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.430300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.430275 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:12.541690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.541664 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.541832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.541766 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.541885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.541846 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.541885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.541868 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sl8b\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.643354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.643298 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.643354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.643339 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sl8b\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.643520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.643407 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.643634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.643608 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.644167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.644139 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.644263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.644245 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.646015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.645992 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.700524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.700497 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sl8b\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\") pod \"payload-processing-external-model-84f575b468-8ljj8\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:12.990474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:12.990400 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:13.365631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:13.365610 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:13.367238 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:30:13.367214 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2667ad18_00e8_49aa_b346_9b800c102b96.slice/crio-279b9876bf24d3ce120af458895cd86e7eacb35d4e5686387b97d19280b64ac1 WatchSource:0}: Error finding container 279b9876bf24d3ce120af458895cd86e7eacb35d4e5686387b97d19280b64ac1: Status 404 returned error can't find the container with id 279b9876bf24d3ce120af458895cd86e7eacb35d4e5686387b97d19280b64ac1 Nov 02 21:30:14.038060 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:14.038025 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" event={"ID":"2667ad18-00e8-49aa-b346-9b800c102b96","Type":"ContainerStarted","Data":"c0d4d4c3f49d30896a82cfc529c4abe7e6d987a0cc8f9125e9d2dc240fe19a14"} Nov 02 21:30:14.038445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:14.038080 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" event={"ID":"2667ad18-00e8-49aa-b346-9b800c102b96","Type":"ContainerStarted","Data":"279b9876bf24d3ce120af458895cd86e7eacb35d4e5686387b97d19280b64ac1"} Nov 02 21:30:14.038445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:14.038254 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:14.157288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:14.157243 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" podStartSLOduration=2.157231958 podStartE2EDuration="2.157231958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:14.155148404 +0000 UTC m=+1253.703398195" watchObservedRunningTime="2026-11-02 21:30:14.157231958 +0000 UTC m=+1253.705481764" Nov 02 21:30:19.579974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:19.579446 5859 scope.go:122] "RemoveContainer" containerID="b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9" Nov 02 21:30:21.192452 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:21.192419 5859 generic.go:356] "Generic (PLEG): container finished" podID="1633e410-736f-422a-b350-c6bddd09042a" containerID="fbfa66739be5fc7dc57825eb0d0cdcc0cd666b5725fd0240dd6bf66995c85688" exitCode=60 Nov 02 21:30:21.192868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:21.192566 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerDied","Data":"fbfa66739be5fc7dc57825eb0d0cdcc0cd666b5725fd0240dd6bf66995c85688"} Nov 02 21:30:21.192868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:21.192623 5859 scope.go:122] "RemoveContainer" containerID="b07606a392c5f63868ed44d8a8036e33fbfaf3298f807b60de506ce32ce679c9" Nov 02 21:30:21.193040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:21.193023 5859 scope.go:122] "RemoveContainer" containerID="fbfa66739be5fc7dc57825eb0d0cdcc0cd666b5725fd0240dd6bf66995c85688" Nov 02 21:30:21.193313 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:21.193295 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc_odh-ai-gateway-infra(1633e410-736f-422a-b350-c6bddd09042a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" podUID="1633e410-736f-422a-b350-c6bddd09042a" Nov 02 21:30:22.330057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.330017 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:22.517097 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.517072 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:22.583889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.583837 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\" (UniqueName: \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\") pod \"1633e410-736f-422a-b350-c6bddd09042a\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " Nov 02 21:30:22.584022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.583943 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\") pod \"1633e410-736f-422a-b350-c6bddd09042a\" (UID: \"1633e410-736f-422a-b350-c6bddd09042a\") " Nov 02 21:30:22.584289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.584267 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca" pod "1633e410-736f-422a-b350-c6bddd09042a" (UID: "1633e410-736f-422a-b350-c6bddd09042a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:22.584509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.584492 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1633e410-736f-422a-b350-c6bddd09042a-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:22.585889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.585864 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn" pod "1633e410-736f-422a-b350-c6bddd09042a" (UID: "1633e410-736f-422a-b350-c6bddd09042a"). InnerVolumeSpecName "kube-api-access-w2xjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:22.685565 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.685535 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xjn\" (UniqueName: \"kubernetes.io/projected/1633e410-736f-422a-b350-c6bddd09042a-kube-api-access-w2xjn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:22.792288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792260 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 21:30:22.792865 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792852 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.792907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792867 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.792907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792888 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.792907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792893 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.792907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792901 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.792907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.792906 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.793069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.793033 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.793069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.793043 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.793069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.793048 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1633e410-736f-422a-b350-c6bddd09042a" containerName="cleanup" Nov 02 21:30:22.796528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.796511 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.859310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.859243 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 21:30:22.887005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.886980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887047 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5wp\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887076 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887110 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887185 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887261 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887282 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887311 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.887337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.887325 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988218 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988303 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988336 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988368 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988437 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988488 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5wp\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988520 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988740 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988779 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.988967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.988931 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.989080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.989024 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.989237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.989220 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.990506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.990484 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:22.990871 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:22.990850 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:23.045026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.045001 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:23.045116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.045047 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5wp\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:23.070274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.070250 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:30:23.070476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.070454 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" containerID="cri-o://11389928f507aa59b288c46ad3466a23c97f0b138d8c58a8338dc8985d790c69" gracePeriod=30 Nov 02 21:30:23.109560 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.109509 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:23.211527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.211500 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" Nov 02 21:30:23.211823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.211796 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc" event={"ID":"1633e410-736f-422a-b350-c6bddd09042a","Type":"ContainerDied","Data":"79ce58bdf0c822fa407335c6d5ff53e4c97b37aa607eb33059a2a71132a027b8"} Nov 02 21:30:23.211927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.211858 5859 scope.go:122] "RemoveContainer" containerID="fbfa66739be5fc7dc57825eb0d0cdcc0cd666b5725fd0240dd6bf66995c85688" Nov 02 21:30:23.380686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.380610 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:23.383613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.383592 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894250-4wtnc"] Nov 02 21:30:23.508825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.508778 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.509159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.509129 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.509328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.509307 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.519172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:23.517263 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 21:30:24.252637 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:24.252596 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" event={"ID":"bff8d3b8-b702-432d-8965-b5fc4c3072ef","Type":"ContainerStarted","Data":"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c"} Nov 02 21:30:24.252822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:24.252664 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" event={"ID":"bff8d3b8-b702-432d-8965-b5fc4c3072ef","Type":"ContainerStarted","Data":"86545492711e34c3b5f8a8ab3b1bc62d033d6b3c623998ee4f82b72c588fe474"} Nov 02 21:30:24.369688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:24.369641 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podStartSLOduration=2.369627252 podStartE2EDuration="2.369627252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:24.36834208 +0000 UTC m=+1263.916591881" watchObservedRunningTime="2026-11-02 21:30:24.369627252 +0000 UTC m=+1263.917877094" Nov 02 21:30:24.571645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:24.571616 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1633e410-736f-422a-b350-c6bddd09042a" path="/var/lib/kubelet/pods/1633e410-736f-422a-b350-c6bddd09042a/volumes" Nov 02 21:30:25.052046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:25.051931 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:30:25.127975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:25.127450 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:25.130248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:25.130132 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:25.130248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:25.130203 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:25.954632 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:25.954599 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:30:26.059911 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:26.059882 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:26.061124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:26.061096 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:30:26.078573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:26.078552 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:30:26.143985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:26.143629 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:26.143985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:26.143678 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:27.126974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:27.117155 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:27.126974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:27.117226 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:27.955329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:27.955296 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:30:27.955490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:27.955349 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:30:28.127971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.110515 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:28.128488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.128458 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:28.284710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.284680 5859 generic.go:356] "Generic (PLEG): container finished" podID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerID="11389928f507aa59b288c46ad3466a23c97f0b138d8c58a8338dc8985d790c69" exitCode=0 Nov 02 21:30:28.284845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.284755 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" event={"ID":"077d6fc9-3b5a-43d2-849b-61c16d3d2cba","Type":"ContainerDied","Data":"11389928f507aa59b288c46ad3466a23c97f0b138d8c58a8338dc8985d790c69"} Nov 02 21:30:28.382648 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.382620 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:30:28.486645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.486615 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.486810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.486674 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.486810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.486757 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.486944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.486901 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487054 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487090 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487110 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487216 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487057 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:28.487351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487303 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\") pod \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\" (UID: \"077d6fc9-3b5a-43d2-849b-61c16d3d2cba\") " Nov 02 21:30:28.487404 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487144 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:28.487404 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487236 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:28.487502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487411 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:28.487672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487646 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:28.487992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487944 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.487992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.487987 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.488163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.488003 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.488163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.488018 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.488163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.488033 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.489121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.489099 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "kube-api-access-j6wm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:28.489262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.489239 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:30:28.489577 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.489409 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:28.489653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.489614 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy" pod "077d6fc9-3b5a-43d2-849b-61c16d3d2cba" (UID: "077d6fc9-3b5a-43d2-849b-61c16d3d2cba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:28.589634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.589603 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.589745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.589638 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.589745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.589653 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6wm8\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-kube-api-access-j6wm8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.589745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:28.589666 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/077d6fc9-3b5a-43d2-849b-61c16d3d2cba-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:29.127986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.127561 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:29.127986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.127617 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:29.293432 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.293399 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" event={"ID":"077d6fc9-3b5a-43d2-849b-61c16d3d2cba","Type":"ContainerDied","Data":"26e9ecdedc1ed7fdb1bd564859e8de3e3b77bd6054516bc7d85ef0d34aa0377c"} Nov 02 21:30:29.293589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.293460 5859 scope.go:122] "RemoveContainer" containerID="11389928f507aa59b288c46ad3466a23c97f0b138d8c58a8338dc8985d790c69" Nov 02 21:30:29.293589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.293412 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws" Nov 02 21:30:29.509298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.509209 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:30:29.559599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:29.559562 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-5z4ws"] Nov 02 21:30:30.110037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:30.110005 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:30.110204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:30.110054 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:30:30.583149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:30.583109 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" path="/var/lib/kubelet/pods/077d6fc9-3b5a-43d2-849b-61c16d3d2cba/volumes" Nov 02 21:30:31.060571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.060496 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" containerID="cri-o://c0d4d4c3f49d30896a82cfc529c4abe7e6d987a0cc8f9125e9d2dc240fe19a14" gracePeriod=25 Nov 02 21:30:31.078980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.078886 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" podUID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" containerName="payload-processing" containerID="cri-o://a0b2a09b0a465c287bba11920cfa2890fdfc26cdb050159b56a865c0c0b64f31" gracePeriod=25 Nov 02 21:30:31.114275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.114253 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:31.114718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.114697 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:31.115444 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.115426 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 21:30:31.326344 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.326308 5859 generic.go:356] "Generic (PLEG): container finished" podID="2667ad18-00e8-49aa-b346-9b800c102b96" containerID="c0d4d4c3f49d30896a82cfc529c4abe7e6d987a0cc8f9125e9d2dc240fe19a14" exitCode=0 Nov 02 21:30:31.326481 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.326461 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" event={"ID":"2667ad18-00e8-49aa-b346-9b800c102b96","Type":"ContainerDied","Data":"c0d4d4c3f49d30896a82cfc529c4abe7e6d987a0cc8f9125e9d2dc240fe19a14"} Nov 02 21:30:31.329613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.328623 5859 generic.go:356] "Generic (PLEG): container finished" podID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" containerID="a0b2a09b0a465c287bba11920cfa2890fdfc26cdb050159b56a865c0c0b64f31" exitCode=0 Nov 02 21:30:31.329613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.329488 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" event={"ID":"416d4bd7-d4ef-4efc-a61d-14609d6aa05f","Type":"ContainerDied","Data":"a0b2a09b0a465c287bba11920cfa2890fdfc26cdb050159b56a865c0c0b64f31"} Nov 02 21:30:31.356662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.356634 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:31.383214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.383196 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:30:31.446792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.446760 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\") pod \"2667ad18-00e8-49aa-b346-9b800c102b96\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " Nov 02 21:30:31.446992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.446829 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\") pod \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " Nov 02 21:30:31.446992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.446874 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\") pod \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " Nov 02 21:30:31.446992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.446974 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\") pod \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " Nov 02 21:30:31.447160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447024 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\") pod \"2667ad18-00e8-49aa-b346-9b800c102b96\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " Nov 02 21:30:31.447160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447117 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\") pod \"2667ad18-00e8-49aa-b346-9b800c102b96\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " Nov 02 21:30:31.447269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447244 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\") pod \"2667ad18-00e8-49aa-b346-9b800c102b96\" (UID: \"2667ad18-00e8-49aa-b346-9b800c102b96\") " Nov 02 21:30:31.447323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447312 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\") pod \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\" (UID: \"416d4bd7-d4ef-4efc-a61d-14609d6aa05f\") " Nov 02 21:30:31.447421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447359 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume" pod "416d4bd7-d4ef-4efc-a61d-14609d6aa05f" (UID: "416d4bd7-d4ef-4efc-a61d-14609d6aa05f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:31.447421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447367 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume" pod "416d4bd7-d4ef-4efc-a61d-14609d6aa05f" (UID: "416d4bd7-d4ef-4efc-a61d-14609d6aa05f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:31.447666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447467 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume" pod "2667ad18-00e8-49aa-b346-9b800c102b96" (UID: "2667ad18-00e8-49aa-b346-9b800c102b96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:31.447741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.447719 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume" pod "2667ad18-00e8-49aa-b346-9b800c102b96" (UID: "2667ad18-00e8-49aa-b346-9b800c102b96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:31.448229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.448206 5859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-routing-overlay-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.448364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.448348 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.448459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.448445 5859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2667ad18-00e8-49aa-b346-9b800c102b96-routing-overlay-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.448558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.448538 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.449550 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.449520 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials" pod "416d4bd7-d4ef-4efc-a61d-14609d6aa05f" (UID: "416d4bd7-d4ef-4efc-a61d-14609d6aa05f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.449994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.449973 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b" pod "2667ad18-00e8-49aa-b346-9b800c102b96" (UID: "2667ad18-00e8-49aa-b346-9b800c102b96"). InnerVolumeSpecName "kube-api-access-2sl8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.450085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.450071 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials" pod "2667ad18-00e8-49aa-b346-9b800c102b96" (UID: "2667ad18-00e8-49aa-b346-9b800c102b96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.450179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.450161 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv" pod "416d4bd7-d4ef-4efc-a61d-14609d6aa05f" (UID: "416d4bd7-d4ef-4efc-a61d-14609d6aa05f"). InnerVolumeSpecName "kube-api-access-99rjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.550394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.550358 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sl8b\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-kube-api-access-2sl8b\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.550394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.550393 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99rjv\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-kube-api-access-99rjv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.550580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.550409 5859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2667ad18-00e8-49aa-b346-9b800c102b96-provider-credentials\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.550580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:31.550425 5859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/416d4bd7-d4ef-4efc-a61d-14609d6aa05f-provider-credentials\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:32.335629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.335562 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" Nov 02 21:30:32.335629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.335605 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv" event={"ID":"416d4bd7-d4ef-4efc-a61d-14609d6aa05f","Type":"ContainerDied","Data":"1c4a3c8f8b45df29e6f7ccdc0818f90707e9c270ce69e5d1bb5019718d59fc51"} Nov 02 21:30:32.336139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.335666 5859 scope.go:122] "RemoveContainer" containerID="a0b2a09b0a465c287bba11920cfa2890fdfc26cdb050159b56a865c0c0b64f31" Nov 02 21:30:32.337318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.337296 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" Nov 02 21:30:32.337436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.337331 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84f575b468-8ljj8" event={"ID":"2667ad18-00e8-49aa-b346-9b800c102b96","Type":"ContainerDied","Data":"279b9876bf24d3ce120af458895cd86e7eacb35d4e5686387b97d19280b64ac1"} Nov 02 21:30:32.345672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.345654 5859 scope.go:122] "RemoveContainer" containerID="c0d4d4c3f49d30896a82cfc529c4abe7e6d987a0cc8f9125e9d2dc240fe19a14" Nov 02 21:30:32.500535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.500509 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:30:32.550574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.550546 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57d7dc98b6-tgmpv"] Nov 02 21:30:32.571322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.571301 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" path="/var/lib/kubelet/pods/416d4bd7-d4ef-4efc-a61d-14609d6aa05f/volumes" Nov 02 21:30:32.660248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.660194 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:32.662753 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:32.662735 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84f575b468-8ljj8"] Nov 02 21:30:33.491620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.491573 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:30:33.492284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492269 5859 cpu_manager.go:414] "containerMap: removing container" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" Nov 02 21:30:33.492284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492285 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" Nov 02 21:30:33.492371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492319 5859 cpu_manager.go:414] "containerMap: removing container" podUID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" containerName="payload-processing" Nov 02 21:30:33.492371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492324 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" containerName="payload-processing" Nov 02 21:30:33.492371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492336 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" Nov 02 21:30:33.492371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492341 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" Nov 02 21:30:33.492513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492425 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="077d6fc9-3b5a-43d2-849b-61c16d3d2cba" containerName="istio-proxy" Nov 02 21:30:33.492513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492439 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="416d4bd7-d4ef-4efc-a61d-14609d6aa05f" containerName="payload-processing" Nov 02 21:30:33.492513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.492445 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" containerName="payload-processing" Nov 02 21:30:33.497136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.497118 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.549879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.549855 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-70efbbab-serving-cert\"" Nov 02 21:30:33.551401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.551380 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:30:33.573944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.573922 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57zw\" (UniqueName: \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.574082 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.574067 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.574166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.574144 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.574236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.574223 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.675177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.675146 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.675336 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.675186 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.675336 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.675232 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.675466 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:33.675353 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-serving-cert: secret "maas-api-e2e-mt-70efbbab-serving-cert" not found Nov 02 21:30:33.675466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.675374 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57zw\" (UniqueName: \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.675466 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:33.675433 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls podName:ab76a576-ad6f-43c9-9cda-84646104a01f nodeName:}" failed. No retries permitted until 2026-11-02 21:30:34.175411169 +0000 UTC m=+1273.723660929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls") pod "maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" (UID: "ab76a576-ad6f-43c9-9cda-84646104a01f") : secret "maas-api-e2e-mt-70efbbab-serving-cert" not found Nov 02 21:30:33.675899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.675871 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.677649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.677631 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:33.732537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:33.732511 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57zw\" (UniqueName: \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:34.182513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:34.182431 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:34.185033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:34.185010 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hz78f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:34.410553 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:34.410525 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:34.570490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:34.570458 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2667ad18-00e8-49aa-b346-9b800c102b96" path="/var/lib/kubelet/pods/2667ad18-00e8-49aa-b346-9b800c102b96/volumes" Nov 02 21:30:34.784160 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:30:34.784132 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab76a576_ad6f_43c9_9cda_84646104a01f.slice/crio-c11dc8046d2d131db68a07ac1864a8d68dc81c1b4e2c23fabd72319a666d6300 WatchSource:0}: Error finding container c11dc8046d2d131db68a07ac1864a8d68dc81c1b4e2c23fabd72319a666d6300: Status 404 returned error can't find the container with id c11dc8046d2d131db68a07ac1864a8d68dc81c1b4e2c23fabd72319a666d6300 Nov 02 21:30:34.784317 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:34.784295 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:30:35.360740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:35.360707 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" event={"ID":"ab76a576-ad6f-43c9-9cda-84646104a01f","Type":"ContainerStarted","Data":"c11dc8046d2d131db68a07ac1864a8d68dc81c1b4e2c23fabd72319a666d6300"} Nov 02 21:30:36.368755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:36.368721 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" event={"ID":"ab76a576-ad6f-43c9-9cda-84646104a01f","Type":"ContainerStarted","Data":"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec"} Nov 02 21:30:36.369123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:36.368775 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:36.486865 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:36.486808 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" podStartSLOduration=2.45146284 podStartE2EDuration="3.486789131s" totalImagesPullingTime="1.035326291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:36.4853261 +0000 UTC m=+1276.033575881" watchObservedRunningTime="2026-11-02 21:30:36.486789131 +0000 UTC m=+1276.035038912" Nov 02 21:30:40.082768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.082739 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:30:40.086736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.086720 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.152857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.152832 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153007 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.152872 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153010 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddcd\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153052 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153155 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153184 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153236 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153301 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153270 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.153369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.153352 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.194868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.194835 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:30:40.254822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.254799 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.254982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.254853 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddcd\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.254982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.254882 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.254985 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255165 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255097 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255164 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255224 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255280 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255302 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255323 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255388 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255526 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255494 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255556 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.255966 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.255935 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.257398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.257376 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.257515 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.257495 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.311078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.311056 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.311171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.311119 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddcd\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.400650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.400582 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:40.827284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.827240 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:30:40.827442 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:30:40.827404 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95687e4_31e2_4b33_9d00_582688e3fa0e.slice/crio-f2a2c315af903f94fe628b71696ec7436f3445c542035ea2cdee9d73cf9f426e WatchSource:0}: Error finding container f2a2c315af903f94fe628b71696ec7436f3445c542035ea2cdee9d73cf9f426e: Status 404 returned error can't find the container with id f2a2c315af903f94fe628b71696ec7436f3445c542035ea2cdee9d73cf9f426e Nov 02 21:30:40.829704 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.829674 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:40.829797 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.829732 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:40.829797 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:40.829763 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:41.436721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:41.436688 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" event={"ID":"a95687e4-31e2-4b33-9d00-582688e3fa0e","Type":"ContainerStarted","Data":"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95"} Nov 02 21:30:41.437083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:41.436739 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" event={"ID":"a95687e4-31e2-4b33-9d00-582688e3fa0e","Type":"ContainerStarted","Data":"f2a2c315af903f94fe628b71696ec7436f3445c542035ea2cdee9d73cf9f426e"} Nov 02 21:30:41.611783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:41.609682 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podStartSLOduration=1.6096640770000001 podStartE2EDuration="1.609664077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:41.605022605 +0000 UTC m=+1281.153272414" watchObservedRunningTime="2026-11-02 21:30:41.609664077 +0000 UTC m=+1281.157913859" Nov 02 21:30:42.401021 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:42.400986 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:42.402428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:42.402406 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:42.402533 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:42.402459 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:42.434051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:42.434020 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:30:43.401711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:43.401677 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:43.402101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:43.401740 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:44.401193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:44.401162 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:44.401343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:44.401211 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:45.442846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:45.442801 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:45.443436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:45.443386 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:46.410767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:46.410302 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:46.410767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:46.410376 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:46.877976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:46.869927 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.901973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:46.901024 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:30:46.928973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:46.928395 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.073975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.073314 5859 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 21:30:47.163040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.161600 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:30:47.163040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.161663 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.404987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.404436 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:47.404987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.404514 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:47.791979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.790070 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.964068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.952101 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.964068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:47.952733 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:30:48.031974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.031059 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:30:48.031974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.031431 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 21:30:48.056089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.032242 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:30:48.056089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.032297 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:30:48.347186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.313669 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 21:30:48.403672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.403144 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:48.403672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.403202 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:48.806848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.806770 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 21:30:48.993854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:48.993810 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 21:30:49.401963 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:49.401910 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:30:49.402117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:49.401997 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:30:50.938882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:50.938850 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:50.939522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:50.939246 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:50.940051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:50.940030 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:30:54.670229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.670197 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:30:54.675707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.675684 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.752387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.752358 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:30:54.765690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.765656 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.765840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.765805 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.765904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.765879 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766012 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766053 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766176 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766156 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766193 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxv9\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766316 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.766405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.766390 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.807008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.806987 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d58d18-openshift-default-dockercfg-6dd47\"" Nov 02 21:30:54.867839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.867797 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.867900 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868006 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868080 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868136 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868163 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868208 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868234 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxv9\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868324 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.868785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868719 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.870110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.868807 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.872781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.872753 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.873003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.872930 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.873184 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.872690 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.880560 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.880490 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.880945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.880906 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.934987 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.934877 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.940555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.940526 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxv9\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:54.990143 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:54.990113 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:55.428396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.426014 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:30:55.430892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.430862 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:55.431012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.430923 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:55.431012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.430982 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:55.946324 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:30:55.946283 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:30:55.974839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.974814 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerStarted","Data":"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b"} Nov 02 21:30:55.974985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.974858 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerStarted","Data":"eae4f7624937c21c71af59a2fc114834f80859c1f594d41e621f6418caa50502"} Nov 02 21:30:55.991141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:55.991119 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:56.093195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:56.093151 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podStartSLOduration=2.093137837 podStartE2EDuration="2.093137837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:56.090310051 +0000 UTC m=+1295.638559848" watchObservedRunningTime="2026-11-02 21:30:56.093137837 +0000 UTC m=+1295.641387617" Nov 02 21:30:56.995583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:56.995548 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:57.989126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:57.989094 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:30:57.990051 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:30:57.990031 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:31:02.743742 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.741124 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:31:02.758549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.758517 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:02.801439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.801405 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:31:02.848368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.848332 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d58d18\"" Nov 02 21:31:02.848529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.848454 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d58d18-dockercfg-txjv8\"" Nov 02 21:31:02.889381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.889350 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:02.889579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.889561 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmks\" (UniqueName: \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:02.990900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.990862 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmks\" (UniqueName: \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:02.991110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.991081 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:02.991766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:02.991745 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:03.049588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:03.049516 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmks\" (UniqueName: \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\") pod \"payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:03.095439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:03.095395 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:03.476798 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:03.476769 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:31:03.478016 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:31:03.477988 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc2f1ee_7639_4ecb_8325_dd7877c97595.slice/crio-adb745c88fa596a1f591e0b24b7b048ac84679db445ef5ff6f589dc08c068be9 WatchSource:0}: Error finding container adb745c88fa596a1f591e0b24b7b048ac84679db445ef5ff6f589dc08c068be9: Status 404 returned error can't find the container with id adb745c88fa596a1f591e0b24b7b048ac84679db445ef5ff6f589dc08c068be9 Nov 02 21:31:04.028036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:04.028003 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" event={"ID":"6fc2f1ee-7639-4ecb-8325-dd7877c97595","Type":"ContainerStarted","Data":"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf"} Nov 02 21:31:04.028474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:04.028059 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" event={"ID":"6fc2f1ee-7639-4ecb-8325-dd7877c97595","Type":"ContainerStarted","Data":"adb745c88fa596a1f591e0b24b7b048ac84679db445ef5ff6f589dc08c068be9"} Nov 02 21:31:04.028474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:04.028234 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:04.146296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:04.146231 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" podStartSLOduration=2.146211689 podStartE2EDuration="2.146211689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:04.143547098 +0000 UTC m=+1303.691796906" watchObservedRunningTime="2026-11-02 21:31:04.146211689 +0000 UTC m=+1303.694461470" Nov 02 21:31:12.292854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.292813 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:31:12.297625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.297603 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.351083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.351052 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:31:12.398165 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.398130 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-03ece2\"" Nov 02 21:31:12.398332 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.398227 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-03ece2-dockercfg-ndzn2\"" Nov 02 21:31:12.477001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.470464 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.486710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.480168 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2stt\" (UniqueName: \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.486710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.480187 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:31:12.489136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.489111 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.539485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.539455 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:31:12.581634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.581599 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.581903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.581875 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.582136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.582112 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2stt\" (UniqueName: \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.582215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.582204 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cnf\" (UniqueName: \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.583152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.583132 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.640669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.640638 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2stt\" (UniqueName: \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\") pod \"payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:12.683785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.683757 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cnf\" (UniqueName: \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.683939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.683861 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.684585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.684566 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.740903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.740870 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cnf\" (UniqueName: \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\") pod \"payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.830080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.830050 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:12.916411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:12.916372 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:13.204903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:13.204873 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:31:13.206638 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:31:13.206611 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccf7da5_873e_4086_bf13_d4818a5f535b.slice/crio-33ada2d8f5554150b1511a506683232faf25591a6b1d530c7c85348e7365d009 WatchSource:0}: Error finding container 33ada2d8f5554150b1511a506683232faf25591a6b1d530c7c85348e7365d009: Status 404 returned error can't find the container with id 33ada2d8f5554150b1511a506683232faf25591a6b1d530c7c85348e7365d009 Nov 02 21:31:13.314717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:13.314690 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:31:13.315188 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:31:13.315169 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01ec63f_6f17_49fd_87fd_667173659e42.slice/crio-b27da0dd53e966be10162337032eb030671e5e96d9325791483f1cd3ec491c86 WatchSource:0}: Error finding container b27da0dd53e966be10162337032eb030671e5e96d9325791483f1cd3ec491c86: Status 404 returned error can't find the container with id b27da0dd53e966be10162337032eb030671e5e96d9325791483f1cd3ec491c86 Nov 02 21:31:14.124760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.124719 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" event={"ID":"c01ec63f-6f17-49fd-87fd-667173659e42","Type":"ContainerStarted","Data":"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba"} Nov 02 21:31:14.124983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.124790 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:14.124983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.124816 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" event={"ID":"c01ec63f-6f17-49fd-87fd-667173659e42","Type":"ContainerStarted","Data":"b27da0dd53e966be10162337032eb030671e5e96d9325791483f1cd3ec491c86"} Nov 02 21:31:14.126300 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.126276 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" event={"ID":"0ccf7da5-873e-4086-bf13-d4818a5f535b","Type":"ContainerStarted","Data":"1cd31e48c56a74b0eaff0800e02aa52ea294ce59e584bfaa1a85c49371b68914"} Nov 02 21:31:14.126399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.126315 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" event={"ID":"0ccf7da5-873e-4086-bf13-d4818a5f535b","Type":"ContainerStarted","Data":"33ada2d8f5554150b1511a506683232faf25591a6b1d530c7c85348e7365d009"} Nov 02 21:31:14.126399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.126370 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:14.242113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.242066 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" podStartSLOduration=2.242054361 podStartE2EDuration="2.242054361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:14.240233854 +0000 UTC m=+1313.788483655" watchObservedRunningTime="2026-11-02 21:31:14.242054361 +0000 UTC m=+1313.790304142" Nov 02 21:31:14.361883 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:14.361826 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" podStartSLOduration=2.361809394 podStartE2EDuration="2.361809394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:14.357770687 +0000 UTC m=+1313.906020489" watchObservedRunningTime="2026-11-02 21:31:14.361809394 +0000 UTC m=+1313.910059177" Nov 02 21:31:15.037248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:15.037221 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:31:25.136395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:25.136365 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:31:25.137005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:25.136440 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:31:25.946657 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:31:25.946616 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:31:27.966414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:27.966374 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:31:27.972458 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:27.972438 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.026437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.026403 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:31:28.072942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.072907 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:31:28.072942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.072919 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:31:28.072942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.072922 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:31:28.073211 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.073025 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n4jrm\"" Nov 02 21:31:28.085764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.085734 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.085915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.085797 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.085915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.085899 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.086055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.086041 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.086094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.086063 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbhs\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.086127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.086096 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.086269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.086248 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187184 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187150 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187358 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187190 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbhs\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187377 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187457 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187440 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187604 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187660 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.187762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.187704 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.188244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.188218 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.189659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.189629 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.189824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.189809 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.189887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.189826 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.189941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.189895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.245369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.245286 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.245369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.245344 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbhs\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8x99k\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.285456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.285427 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:28.713087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:28.713049 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:31:28.751568 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:31:28.751537 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb412a8_f9df_43b5_9416_a0ce1b96a96f.slice/crio-0ea8c738979fea0b178b229f2b532583088b0af2b33ab1693b7828e221875737 WatchSource:0}: Error finding container 0ea8c738979fea0b178b229f2b532583088b0af2b33ab1693b7828e221875737: Status 404 returned error can't find the container with id 0ea8c738979fea0b178b229f2b532583088b0af2b33ab1693b7828e221875737 Nov 02 21:31:29.225568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:29.225530 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" event={"ID":"7eb412a8-f9df-43b5-9416-a0ce1b96a96f","Type":"ContainerStarted","Data":"0ea8c738979fea0b178b229f2b532583088b0af2b33ab1693b7828e221875737"} Nov 02 21:31:32.919434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:32.919396 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:32.919749 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:32.919464 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:33.255068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:33.254983 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" event={"ID":"7eb412a8-f9df-43b5-9416-a0ce1b96a96f","Type":"ContainerStarted","Data":"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d"} Nov 02 21:31:33.255278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:33.255258 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:33.256560 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:33.256538 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8x99k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:31:33.256679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:33.256577 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:31:33.462385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:33.462338 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" podStartSLOduration=2.2978256630000002 podStartE2EDuration="6.462325541s" totalImagesPullingTime="4.164499878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:33.422222638 +0000 UTC m=+1332.970472418" watchObservedRunningTime="2026-11-02 21:31:33.462325541 +0000 UTC m=+1333.010575304" Nov 02 21:31:34.267902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:34.267424 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8x99k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:31:34.267902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:34.267493 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:31:37.265091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:37.265058 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:31:38.652920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.652878 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:31:38.657405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.657381 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.710031 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.710006 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\"" Nov 02 21:31:38.712620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712595 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.712714 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712651 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.712761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712744 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.712793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712767 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.712880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712868 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.712942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.712930 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll447\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.713077 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.713061 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.713130 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.713095 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.713235 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.713221 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.765726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.765696 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:31:38.814185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814162 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814245 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll447\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814298 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814316 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814411 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814458 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814506 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814532 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814672 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814778 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814832 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.814939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814920 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.815002 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.814920 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.816577 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.816557 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.816849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.816830 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.871781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.871754 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.872004 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.871988 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll447\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:38.970724 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:38.970673 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:39.398375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:39.398343 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:31:39.398647 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:31:39.398623 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb477b6e7_5c7e_4e33_ac12_0b02bc5de97f.slice/crio-0e00ef50fc9d8a897de4ba0532e46173ab936b3d6a8059f6980962f382bb049d WatchSource:0}: Error finding container 0e00ef50fc9d8a897de4ba0532e46173ab936b3d6a8059f6980962f382bb049d: Status 404 returned error can't find the container with id 0e00ef50fc9d8a897de4ba0532e46173ab936b3d6a8059f6980962f382bb049d Nov 02 21:31:39.401308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:39.401275 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:39.401403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:39.401337 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:39.401403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:39.401365 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:40.354049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.347163 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" event={"ID":"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f","Type":"ContainerStarted","Data":"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb"} Nov 02 21:31:40.354049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.347276 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" event={"ID":"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f","Type":"ContainerStarted","Data":"0e00ef50fc9d8a897de4ba0532e46173ab936b3d6a8059f6980962f382bb049d"} Nov 02 21:31:40.576595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.576544 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podStartSLOduration=2.576530778 podStartE2EDuration="2.576530778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:40.523448119 +0000 UTC m=+1340.071697926" watchObservedRunningTime="2026-11-02 21:31:40.576530778 +0000 UTC m=+1340.124780558" Nov 02 21:31:40.971460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.971428 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:40.973023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.973000 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:31:40.973149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:40.973046 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 21:31:41.971991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:41.971944 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:31:41.972447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:41.972019 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 21:31:42.971252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:42.971219 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:31:42.971435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:42.971267 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 21:31:43.972027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:43.971990 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:31:43.972430 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:43.972036 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 21:31:44.972050 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:44.972018 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:31:44.972417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:44.972072 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 21:31:46.228409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:46.228377 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:46.228883 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:46.228765 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:46.229588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:31:46.229566 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:31:55.945071 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:31:55.945036 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:32:12.580348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.580316 5859 generic.go:356] "Generic (PLEG): container finished" podID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerID="cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b" exitCode=0 Nov 02 21:32:12.580774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.580382 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerDied","Data":"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b"} Nov 02 21:32:12.580774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.580742 5859 scope.go:122] "RemoveContainer" containerID="cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b" Nov 02 21:32:12.582211 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.582183 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:12.582315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.582248 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:12.582315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.582278 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:12.989723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:12.989650 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:32:13.588586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:13.588551 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerStarted","Data":"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2"} Nov 02 21:32:13.990342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:13.990265 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:32:13.991284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:13.991256 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:32:13.991375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:13.991310 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:32:14.990541 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:14.990511 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:32:14.990925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:14.990554 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:32:15.990571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:15.990538 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:32:15.990929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:15.990590 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:32:16.991356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:16.991320 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:32:16.991725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:16.991386 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:32:17.990687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:17.990656 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:32:17.990851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:17.990705 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:32:18.995170 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:18.995135 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:32:18.995595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:18.995557 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:32:18.996178 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:18.996159 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:32:25.996746 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:32:25.996682 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:32:40.559237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.559201 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:32:40.568221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.568194 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.664037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.663926 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.664651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.664430 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mpr\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.664651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.664493 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.664651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.664552 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.665107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.665042 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.665107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.665081 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.666609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.666132 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.666609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.666240 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.666609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.666314 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.676544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.676519 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.768931 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.768991 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769071 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769113 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769159 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769248 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769393 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mpr\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769403 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769464 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.769535 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.770206 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.770375 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.771357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.770626 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.772195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.771758 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.777152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.775523 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.777847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.777821 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.837339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.837301 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.837536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.837518 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mpr\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:40.883026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:40.883000 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:41.309142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.309110 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:32:41.309248 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:32:41.309178 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8164fb9_c683_42ec_956f_094cec96fc34.slice/crio-93940e803cfa62f30053a66b3bd33c810d8ff2844252397e4cc603d30bed1b97 WatchSource:0}: Error finding container 93940e803cfa62f30053a66b3bd33c810d8ff2844252397e4cc603d30bed1b97: Status 404 returned error can't find the container with id 93940e803cfa62f30053a66b3bd33c810d8ff2844252397e4cc603d30bed1b97 Nov 02 21:32:41.311486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.311457 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.311562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.311517 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.311562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.311542 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.865437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.865401 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" event={"ID":"e8164fb9-c683-42ec-956f-094cec96fc34","Type":"ContainerStarted","Data":"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700"} Nov 02 21:32:41.865818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.865478 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" event={"ID":"e8164fb9-c683-42ec-956f-094cec96fc34","Type":"ContainerStarted","Data":"93940e803cfa62f30053a66b3bd33c810d8ff2844252397e4cc603d30bed1b97"} Nov 02 21:32:41.883228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:41.883204 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:42.884599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:42.884562 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:32:42.885015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:42.884612 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 21:32:43.883687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:43.883656 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:32:43.883868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:43.883702 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 21:32:44.883996 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:44.883930 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:32:44.884486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:44.884003 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 21:32:45.883494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:45.883463 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:32:45.883661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:45.883511 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 21:32:46.883687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:46.883651 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 21:32:46.884085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:46.883707 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 21:32:47.887997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:47.887966 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:47.888397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:47.888274 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:47.888874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:47.888857 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:32:48.058634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:48.058590 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podStartSLOduration=8.058577455 podStartE2EDuration="8.058577455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:41.992284333 +0000 UTC m=+1401.540534155" watchObservedRunningTime="2026-11-02 21:32:48.058577455 +0000 UTC m=+1407.606827300" Nov 02 21:32:55.661628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.661596 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:32:55.665958 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.665934 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.774189 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.774158 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:32:55.817060 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817031 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817081 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817174 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817362 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817342 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817456 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6s7r\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817603 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817534 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817603 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817587 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817616 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.817738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.817712 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918295 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918359 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6s7r\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918390 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918414 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918446 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918574 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918723 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918627 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918742 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918768 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.918831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918794 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.919010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.918850 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.919136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.919115 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.919240 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.919221 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.919424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.919404 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.920848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.920826 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.921222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.921206 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.944384 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:32:55.944355 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:32:55.975061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.975035 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.975154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.975107 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6s7r\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:55.979876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:55.979852 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:56.412396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.412370 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:32:56.413543 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:32:56.413519 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12214cb_123b_42e1_be01_a2984b425fbe.slice/crio-bdbd749a5f503897ec1e4632d06925068dde1aca0a0505e7d1ac1c3ebbc6059f WatchSource:0}: Error finding container bdbd749a5f503897ec1e4632d06925068dde1aca0a0505e7d1ac1c3ebbc6059f: Status 404 returned error can't find the container with id bdbd749a5f503897ec1e4632d06925068dde1aca0a0505e7d1ac1c3ebbc6059f Nov 02 21:32:56.416101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.416063 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:56.416198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.416150 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:56.416263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.416197 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:56.971944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.971910 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" event={"ID":"f12214cb-123b-42e1-be01-a2984b425fbe","Type":"ContainerStarted","Data":"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b"} Nov 02 21:32:56.972340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.971987 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" event={"ID":"f12214cb-123b-42e1-be01-a2984b425fbe","Type":"ContainerStarted","Data":"bdbd749a5f503897ec1e4632d06925068dde1aca0a0505e7d1ac1c3ebbc6059f"} Nov 02 21:32:56.980274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:56.980252 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:32:57.093381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:57.093313 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podStartSLOduration=2.093299357 podStartE2EDuration="2.093299357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:57.090447676 +0000 UTC m=+1416.638697483" watchObservedRunningTime="2026-11-02 21:32:57.093299357 +0000 UTC m=+1416.641549157" Nov 02 21:32:57.981747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:57.981711 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:32:57.982191 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:57.981766 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:32:58.980237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:58.980209 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:32:58.980408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:58.980264 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:32:59.980761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:59.980731 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:32:59.981230 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:32:59.981204 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:33:00.980748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:00.980714 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:00.980929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:00.980777 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:33:01.980580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:01.980544 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:01.980757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:01.980597 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:33:02.984555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:02.984524 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:33:02.985052 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:02.984862 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:33:02.985586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:02.985565 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:33:25.944646 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:33:25.944604 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:33:27.286556 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.286525 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:33:27.291163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.291144 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.350741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.350708 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:33:27.401343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401308 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401453 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401482 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401652 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401690 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401714 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401813 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401843 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.401880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.401864 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hd8r\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.436143 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.436082 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-682116-openshift-default-dockercfg-4m5jl\"" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.503971 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504146 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504191 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504324 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504360 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504390 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504442 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504475 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.504507 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hd8r\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.505972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.505560 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.506655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.506014 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.508997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.506768 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.508997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.507119 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.508997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.507571 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.510270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.510248 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.512618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.512127 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.579578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.579547 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.588573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.588500 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hd8r\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\") pod \"e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:27.605182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:27.605154 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:28.039830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.039802 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:33:28.040308 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:33:28.040287 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode325eb51_8b55_4d3b_99e6_2327b3031b7c.slice/crio-3b9ebb3e23745fbebeb3e32b4f71d9225da8475949aac652e354de455ee4049e WatchSource:0}: Error finding container 3b9ebb3e23745fbebeb3e32b4f71d9225da8475949aac652e354de455ee4049e: Status 404 returned error can't find the container with id 3b9ebb3e23745fbebeb3e32b4f71d9225da8475949aac652e354de455ee4049e Nov 02 21:33:28.042691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.042662 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:28.042762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.042725 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:28.042762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.042754 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:28.197971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.197924 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerStarted","Data":"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc"} Nov 02 21:33:28.198146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.198016 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerStarted","Data":"3b9ebb3e23745fbebeb3e32b4f71d9225da8475949aac652e354de455ee4049e"} Nov 02 21:33:28.316178 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.315841 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podStartSLOduration=1.315824944 podStartE2EDuration="1.315824944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:28.314443346 +0000 UTC m=+1447.862693154" watchObservedRunningTime="2026-11-02 21:33:28.315824944 +0000 UTC m=+1447.864074729" Nov 02 21:33:28.606205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:28.606123 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:29.610340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:29.610309 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:30.216394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:30.216061 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:30.217304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:30.217281 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:33:48.157380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.157342 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:33:48.161908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.161885 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.193583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.193562 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.193684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.193621 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.193732 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.193711 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.193839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.193826 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xlsq\" (UniqueName: \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.215712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.215689 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:33:48.262485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.262450 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bd96ef-serving-cert\"" Nov 02 21:33:48.295418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.295394 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xlsq\" (UniqueName: \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.295524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.295509 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.295570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.295562 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.295657 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:33:48.295637 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-serving-cert: secret "maas-api-e2e-api-b-bd96ef-serving-cert" not found Nov 02 21:33:48.295708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.295664 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.295744 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:33:48.295718 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls podName:da006833-3684-4f85-bc40-d6ae795c5bc1 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:48.795702392 +0000 UTC m=+1468.343952167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls") pod "maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" (UID: "da006833-3684-4f85-bc40-d6ae795c5bc1") : secret "maas-api-e2e-api-b-bd96ef-serving-cert" not found Nov 02 21:33:48.296162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.296142 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.297965 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.297937 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.354850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.354817 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xlsq\" (UniqueName: \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.805108 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.805077 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:48.807572 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:48.807547 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") pod \"maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:49.076914 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:49.076891 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:49.458017 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:49.457988 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:33:49.458467 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:33:49.458441 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda006833_3684_4f85_bc40_d6ae795c5bc1.slice/crio-a8ff746a4dd7e27134684f3c785b82a93438fc5df69ae3fb79f7538fc7fab036 WatchSource:0}: Error finding container a8ff746a4dd7e27134684f3c785b82a93438fc5df69ae3fb79f7538fc7fab036: Status 404 returned error can't find the container with id a8ff746a4dd7e27134684f3c785b82a93438fc5df69ae3fb79f7538fc7fab036 Nov 02 21:33:50.358456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:50.358420 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" event={"ID":"da006833-3684-4f85-bc40-d6ae795c5bc1","Type":"ContainerStarted","Data":"a8ff746a4dd7e27134684f3c785b82a93438fc5df69ae3fb79f7538fc7fab036"} Nov 02 21:33:51.373995 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:51.373935 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" event={"ID":"da006833-3684-4f85-bc40-d6ae795c5bc1","Type":"ContainerStarted","Data":"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6"} Nov 02 21:33:51.384810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:51.384773 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:33:51.506116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:51.506064 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" podStartSLOduration=2.590021149 podStartE2EDuration="3.506049444s" totalImagesPullingTime="916.028295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:51.503290673 +0000 UTC m=+1471.051540466" watchObservedRunningTime="2026-11-02 21:33:51.506049444 +0000 UTC m=+1471.054299224" Nov 02 21:33:55.945069 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:33:55.945012 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:33:58.399663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:33:58.399633 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:34:25.944870 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:34:25.944834 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:34:43.101650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:43.101616 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:34:48.102721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.102683 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" podUID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" containerName="payload-processing" containerID="cri-o://4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf" gracePeriod=25 Nov 02 21:34:48.372612 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.372557 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:34:48.433910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.433887 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\") pod \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " Nov 02 21:34:48.434081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.433978 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\" (UniqueName: \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\") pod \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\" (UID: \"6fc2f1ee-7639-4ecb-8325-dd7877c97595\") " Nov 02 21:34:48.434247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.434225 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume" pod "6fc2f1ee-7639-4ecb-8325-dd7877c97595" (UID: "6fc2f1ee-7639-4ecb-8325-dd7877c97595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:48.434602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.434584 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc2f1ee-7639-4ecb-8325-dd7877c97595-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:48.435844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.435825 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks" pod "6fc2f1ee-7639-4ecb-8325-dd7877c97595" (UID: "6fc2f1ee-7639-4ecb-8325-dd7877c97595"). InnerVolumeSpecName "kube-api-access-4pmks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:48.536180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.536150 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmks\" (UniqueName: \"kubernetes.io/projected/6fc2f1ee-7639-4ecb-8325-dd7877c97595-kube-api-access-4pmks\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:48.782703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.782620 5859 generic.go:356] "Generic (PLEG): container finished" podID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" containerID="4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf" exitCode=0 Nov 02 21:34:48.782703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.782664 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" event={"ID":"6fc2f1ee-7639-4ecb-8325-dd7877c97595","Type":"ContainerDied","Data":"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf"} Nov 02 21:34:48.782703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.782701 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" event={"ID":"6fc2f1ee-7639-4ecb-8325-dd7877c97595","Type":"ContainerDied","Data":"adb745c88fa596a1f591e0b24b7b048ac84679db445ef5ff6f589dc08c068be9"} Nov 02 21:34:48.782914 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.782720 5859 scope.go:122] "RemoveContainer" containerID="4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf" Nov 02 21:34:48.782914 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.782664 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m" Nov 02 21:34:48.792781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.792761 5859 scope.go:122] "RemoveContainer" containerID="4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf" Nov 02 21:34:48.793073 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:34:48.793043 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf\": container with ID starting with 4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf not found: ID does not exist" containerID="4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf" Nov 02 21:34:48.793176 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.793080 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf"} err="failed to get container status \"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf\": rpc error: code = NotFound desc = could not find container \"4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf\": container with ID starting with 4ea16734b2bd119d7c37ca282c5dcd59fba63ab4aeae3cbbb4899dbb379e12bf not found: ID does not exist" Nov 02 21:34:48.952540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.952513 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:34:48.958054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:48.958032 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d58d18-75dddf5b7f-5wg7m"] Nov 02 21:34:50.571646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:34:50.571615 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" path="/var/lib/kubelet/pods/6fc2f1ee-7639-4ecb-8325-dd7877c97595/volumes" Nov 02 21:34:55.957742 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:34:55.957685 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:35:01.257096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:01.257058 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:35:01.257511 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:01.257155 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:01.822843 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:01.822743 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:35:01.929460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:01.929424 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:35:02.900119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.900082 5859 generic.go:356] "Generic (PLEG): container finished" podID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerID="9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc" exitCode=0 Nov 02 21:35:02.900566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.900156 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerDied","Data":"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc"} Nov 02 21:35:02.900566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.900503 5859 scope.go:122] "RemoveContainer" containerID="9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc" Nov 02 21:35:02.901872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.901848 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:02.901936 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.901908 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:02.901997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:02.901937 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:03.934870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:03.934837 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerStarted","Data":"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022"} Nov 02 21:35:04.627975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:04.627839 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:35:04.627975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:04.627966 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 21:35:04.628238 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:04.628023 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 21:35:05.607329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:05.607293 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 21:35:05.607897 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:05.607869 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 21:35:06.633976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.629559 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 21:35:06.633976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.629631 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 21:35:06.828972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.825044 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" podUID="0ccf7da5-873e-4086-bf13-d4818a5f535b" containerName="payload-processing" containerID="cri-o://1cd31e48c56a74b0eaff0800e02aa52ea294ce59e584bfaa1a85c49371b68914" gracePeriod=25 Nov 02 21:35:06.930128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.930037 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" podUID="c01ec63f-6f17-49fd-87fd-667173659e42" containerName="payload-pre-processing" containerID="cri-o://b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba" gracePeriod=25 Nov 02 21:35:06.957855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.957830 5859 generic.go:356] "Generic (PLEG): container finished" podID="0ccf7da5-873e-4086-bf13-d4818a5f535b" containerID="1cd31e48c56a74b0eaff0800e02aa52ea294ce59e584bfaa1a85c49371b68914" exitCode=0 Nov 02 21:35:06.957988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:06.957901 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" event={"ID":"0ccf7da5-873e-4086-bf13-d4818a5f535b","Type":"ContainerDied","Data":"1cd31e48c56a74b0eaff0800e02aa52ea294ce59e584bfaa1a85c49371b68914"} Nov 02 21:35:07.389161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.389046 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:35:07.463868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.463840 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:35:07.480129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480098 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\" (UniqueName: \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\") pod \"0ccf7da5-873e-4086-bf13-d4818a5f535b\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " Nov 02 21:35:07.480247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480208 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\" (UniqueName: \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\") pod \"c01ec63f-6f17-49fd-87fd-667173659e42\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " Nov 02 21:35:07.480247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480244 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\") pod \"0ccf7da5-873e-4086-bf13-d4818a5f535b\" (UID: \"0ccf7da5-873e-4086-bf13-d4818a5f535b\") " Nov 02 21:35:07.480350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480293 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\") pod \"c01ec63f-6f17-49fd-87fd-667173659e42\" (UID: \"c01ec63f-6f17-49fd-87fd-667173659e42\") " Nov 02 21:35:07.480637 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480614 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume" pod "0ccf7da5-873e-4086-bf13-d4818a5f535b" (UID: "0ccf7da5-873e-4086-bf13-d4818a5f535b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:07.480756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480733 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume" pod "c01ec63f-6f17-49fd-87fd-667173659e42" (UID: "c01ec63f-6f17-49fd-87fd-667173659e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:07.480820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.480806 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccf7da5-873e-4086-bf13-d4818a5f535b-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.482270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.482252 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt" pod "c01ec63f-6f17-49fd-87fd-667173659e42" (UID: "c01ec63f-6f17-49fd-87fd-667173659e42"). InnerVolumeSpecName "kube-api-access-f2stt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:07.482330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.482278 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf" pod "0ccf7da5-873e-4086-bf13-d4818a5f535b" (UID: "0ccf7da5-873e-4086-bf13-d4818a5f535b"). InnerVolumeSpecName "kube-api-access-l8cnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:07.594983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.594246 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2stt\" (UniqueName: \"kubernetes.io/projected/c01ec63f-6f17-49fd-87fd-667173659e42-kube-api-access-f2stt\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.594983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.594282 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01ec63f-6f17-49fd-87fd-667173659e42-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.594983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.594305 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cnf\" (UniqueName: \"kubernetes.io/projected/0ccf7da5-873e-4086-bf13-d4818a5f535b-kube-api-access-l8cnf\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.610976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.610553 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 21:35:07.610976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.610631 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 21:35:07.965762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.965731 5859 generic.go:356] "Generic (PLEG): container finished" podID="c01ec63f-6f17-49fd-87fd-667173659e42" containerID="b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba" exitCode=0 Nov 02 21:35:07.966205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.965786 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" Nov 02 21:35:07.966205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.965813 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" event={"ID":"c01ec63f-6f17-49fd-87fd-667173659e42","Type":"ContainerDied","Data":"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba"} Nov 02 21:35:07.966205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.965861 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw" event={"ID":"c01ec63f-6f17-49fd-87fd-667173659e42","Type":"ContainerDied","Data":"b27da0dd53e966be10162337032eb030671e5e96d9325791483f1cd3ec491c86"} Nov 02 21:35:07.966205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.965885 5859 scope.go:122] "RemoveContainer" containerID="b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba" Nov 02 21:35:07.967651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.967628 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" Nov 02 21:35:07.967771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.967649 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9" event={"ID":"0ccf7da5-873e-4086-bf13-d4818a5f535b","Type":"ContainerDied","Data":"33ada2d8f5554150b1511a506683232faf25591a6b1d530c7c85348e7365d009"} Nov 02 21:35:07.978324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.978306 5859 scope.go:122] "RemoveContainer" containerID="b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba" Nov 02 21:35:07.978566 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:07.978547 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba\": container with ID starting with b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba not found: ID does not exist" containerID="b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba" Nov 02 21:35:07.978621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.978572 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba"} err="failed to get container status \"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba\": rpc error: code = NotFound desc = could not find container \"b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba\": container with ID starting with b69327e5582a89a32a4f3da75cde00d86b9898ac4c5ea286df3094f8313073ba not found: ID does not exist" Nov 02 21:35:07.978621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:07.978588 5859 scope.go:122] "RemoveContainer" containerID="1cd31e48c56a74b0eaff0800e02aa52ea294ce59e584bfaa1a85c49371b68914" Nov 02 21:35:08.142197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.142063 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:35:08.192000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.191977 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03ece2-f8f48c879-f9lqw"] Nov 02 21:35:08.255661 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.255631 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:35:08.258635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.258612 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03ece2-6dfd54447f-58ht9"] Nov 02 21:35:08.572634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.572601 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccf7da5-873e-4086-bf13-d4818a5f535b" path="/var/lib/kubelet/pods/0ccf7da5-873e-4086-bf13-d4818a5f535b/volumes" Nov 02 21:35:08.573075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.573061 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01ec63f-6f17-49fd-87fd-667173659e42" path="/var/lib/kubelet/pods/c01ec63f-6f17-49fd-87fd-667173659e42/volumes" Nov 02 21:35:08.606977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.606283 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 21:35:08.606977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:08.606347 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 21:35:09.609881 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:09.609852 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:35:09.610353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:09.610302 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:35:09.610917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:09.610898 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:35:14.973367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:14.973334 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:35:14.973872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:14.973535 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" containerID="cri-o://a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2" gracePeriod=30 Nov 02 21:35:14.979976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:14.979937 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:35:14.980182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:14.980162 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" containerID="cri-o://6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700" gracePeriod=30 Nov 02 21:35:15.017302 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:15.017270 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:35:15.017480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:15.017458 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" containerID="cri-o://9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b" gracePeriod=30 Nov 02 21:35:17.890660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:17.890620 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:17.891049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:17.890696 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:17.987138 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:17.987107 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:17.987272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:17.987167 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:18.997292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:18.997250 5859 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:18.997646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:18.997306 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:20.329778 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.329749 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:35:20.349350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.349329 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:35:20.353285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.353263 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:35:20.401130 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401108 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401156 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401205 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401243 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401297 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401322 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401466 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401479 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.401546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401506 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401517 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.401706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401554 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\") pod \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\" (UID: \"63501607-b967-47e3-8aef-6f8ac2b9cd8b\") " Nov 02 21:35:20.401706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401577 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.401706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401594 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.401875 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.401846 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.402308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.402286 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.402308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.402308 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.402486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.402318 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.402486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.402328 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.402486 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.402339 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.403470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.403449 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.403569 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.403554 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.403862 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.403837 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "kube-api-access-7rxv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.403928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.403872 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token" pod "63501607-b967-47e3-8aef-6f8ac2b9cd8b" (UID: "63501607-b967-47e3-8aef-6f8ac2b9cd8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.503607 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503579 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.503719 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503613 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.503719 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503643 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.503806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503719 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.503806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503777 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.503909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503811 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.503909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503870 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.503933 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504022 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504057 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504051 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.504301 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504228 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.504357 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504336 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.504481 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504461 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.504540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504462 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.504540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504527 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504617 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504652 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504699 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\") pod \"e8164fb9-c683-42ec-956f-094cec96fc34\" (UID: \"e8164fb9-c683-42ec-956f-094cec96fc34\") " Nov 02 21:35:20.504756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504707 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.504756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504744 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.504919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504801 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.504919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504838 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\") pod \"f12214cb-123b-42e1-be01-a2984b425fbe\" (UID: \"f12214cb-123b-42e1-be01-a2984b425fbe\") " Nov 02 21:35:20.505058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.504925 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.505135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505117 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505288 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505556 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505577 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505591 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505605 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505619 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505634 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505647 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63501607-b967-47e3-8aef-6f8ac2b9cd8b-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505664 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12214cb-123b-42e1-be01-a2984b425fbe-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505678 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505693 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505709 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxv9\" (UniqueName: \"kubernetes.io/projected/63501607-b967-47e3-8aef-6f8ac2b9cd8b-kube-api-access-7rxv9\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505722 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.505987 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.506059 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.506036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.506979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.506364 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.506979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.506676 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.506979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.506759 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "kube-api-access-g6mpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.506979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.506764 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.507299 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.507270 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.507621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.507595 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.507937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.507919 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r" pod "f12214cb-123b-42e1-be01-a2984b425fbe" (UID: "f12214cb-123b-42e1-be01-a2984b425fbe"). InnerVolumeSpecName "kube-api-access-p6s7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.508253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.508235 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy" pod "e8164fb9-c683-42ec-956f-094cec96fc34" (UID: "e8164fb9-c683-42ec-956f-094cec96fc34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.607663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607632 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mpr\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-kube-api-access-g6mpr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607666 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8164fb9-c683-42ec-956f-094cec96fc34-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607680 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8164fb9-c683-42ec-956f-094cec96fc34-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607695 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8164fb9-c683-42ec-956f-094cec96fc34-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607707 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6s7r\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-kube-api-access-p6s7r\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607723 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607737 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12214cb-123b-42e1-be01-a2984b425fbe-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607750 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12214cb-123b-42e1-be01-a2984b425fbe-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607763 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8164fb9-c683-42ec-956f-094cec96fc34-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.607796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:20.607776 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12214cb-123b-42e1-be01-a2984b425fbe-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:21.073028 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.072993 5859 generic.go:356] "Generic (PLEG): container finished" podID="f12214cb-123b-42e1-be01-a2984b425fbe" containerID="9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b" exitCode=0 Nov 02 21:35:21.073121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.073080 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" Nov 02 21:35:21.073171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.073090 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" event={"ID":"f12214cb-123b-42e1-be01-a2984b425fbe","Type":"ContainerDied","Data":"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b"} Nov 02 21:35:21.073239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.073226 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t" event={"ID":"f12214cb-123b-42e1-be01-a2984b425fbe","Type":"ContainerDied","Data":"bdbd749a5f503897ec1e4632d06925068dde1aca0a0505e7d1ac1c3ebbc6059f"} Nov 02 21:35:21.073272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.073254 5859 scope.go:122] "RemoveContainer" containerID="9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b" Nov 02 21:35:21.074937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.074911 5859 generic.go:356] "Generic (PLEG): container finished" podID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerID="a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2" exitCode=0 Nov 02 21:35:21.075092 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.074996 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerDied","Data":"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2"} Nov 02 21:35:21.075155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.075106 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" Nov 02 21:35:21.075417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.075394 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2" event={"ID":"63501607-b967-47e3-8aef-6f8ac2b9cd8b","Type":"ContainerDied","Data":"eae4f7624937c21c71af59a2fc114834f80859c1f594d41e621f6418caa50502"} Nov 02 21:35:21.076601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.076577 5859 generic.go:356] "Generic (PLEG): container finished" podID="e8164fb9-c683-42ec-956f-094cec96fc34" containerID="6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700" exitCode=0 Nov 02 21:35:21.076702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.076620 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" Nov 02 21:35:21.076702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.076664 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" event={"ID":"e8164fb9-c683-42ec-956f-094cec96fc34","Type":"ContainerDied","Data":"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700"} Nov 02 21:35:21.076814 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.076710 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878" event={"ID":"e8164fb9-c683-42ec-956f-094cec96fc34","Type":"ContainerDied","Data":"93940e803cfa62f30053a66b3bd33c810d8ff2844252397e4cc603d30bed1b97"} Nov 02 21:35:21.087655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.087633 5859 scope.go:122] "RemoveContainer" containerID="9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b" Nov 02 21:35:21.088238 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:21.088216 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b\": container with ID starting with 9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b not found: ID does not exist" containerID="9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b" Nov 02 21:35:21.088323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.088244 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b"} err="failed to get container status \"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b\": rpc error: code = NotFound desc = could not find container \"9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b\": container with ID starting with 9bed3023d4a30f1171ccb913d9fee9c75922474f32a9359c7f18bf4a0123a79b not found: ID does not exist" Nov 02 21:35:21.088323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.088260 5859 scope.go:122] "RemoveContainer" containerID="a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2" Nov 02 21:35:21.097216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.097196 5859 scope.go:122] "RemoveContainer" containerID="cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b" Nov 02 21:35:21.106644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.106623 5859 scope.go:122] "RemoveContainer" containerID="a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2" Nov 02 21:35:21.106911 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:21.106895 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2\": container with ID starting with a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2 not found: ID does not exist" containerID="a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2" Nov 02 21:35:21.106974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.106916 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2"} err="failed to get container status \"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2\": rpc error: code = NotFound desc = could not find container \"a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2\": container with ID starting with a0c3e2e1539c41e84691cb31ee47845a5006c301911b3e10325a746a57d6f4c2 not found: ID does not exist" Nov 02 21:35:21.106974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.106930 5859 scope.go:122] "RemoveContainer" containerID="cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b" Nov 02 21:35:21.107159 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:21.107139 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b\": container with ID starting with cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b not found: ID does not exist" containerID="cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b" Nov 02 21:35:21.107220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.107168 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b"} err="failed to get container status \"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b\": rpc error: code = NotFound desc = could not find container \"cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b\": container with ID starting with cd5d9db68a1fac3ceef9f6944ad2e2a153c8ce38230a434dfa05ba4402474e3b not found: ID does not exist" Nov 02 21:35:21.107220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.107189 5859 scope.go:122] "RemoveContainer" containerID="6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700" Nov 02 21:35:21.115471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.115453 5859 scope.go:122] "RemoveContainer" containerID="6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700" Nov 02 21:35:21.115721 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:21.115703 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700\": container with ID starting with 6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700 not found: ID does not exist" containerID="6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700" Nov 02 21:35:21.115777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.115726 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700"} err="failed to get container status \"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700\": rpc error: code = NotFound desc = could not find container \"6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700\": container with ID starting with 6876e51729227c0ead983c9c3e9901c5d7e4fdc36903aafc6629aca756962700 not found: ID does not exist" Nov 02 21:35:21.240361 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.240333 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:35:21.289813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.289749 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-mh878"] Nov 02 21:35:21.354465 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.354436 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:35:21.357865 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.357847 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-2cjl2"] Nov 02 21:35:21.468265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.468233 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:35:21.472131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:21.472101 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-qlx2t"] Nov 02 21:35:22.571697 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:22.571662 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" path="/var/lib/kubelet/pods/63501607-b967-47e3-8aef-6f8ac2b9cd8b/volumes" Nov 02 21:35:22.572167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:22.572151 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" path="/var/lib/kubelet/pods/e8164fb9-c683-42ec-956f-094cec96fc34/volumes" Nov 02 21:35:22.572581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:22.572568 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" path="/var/lib/kubelet/pods/f12214cb-123b-42e1-be01-a2984b425fbe/volumes" Nov 02 21:35:25.825041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.825009 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:35:25.825437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.825196 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" containerID="cri-o://7fce3676056670c607412de9fe46a5dd6e0d9b5061991b85f063a58f7afc39f0" gracePeriod=30 Nov 02 21:35:25.875265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.875238 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:35:25.875443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.875424 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" containerID="cri-o://a67da46892b3e388f3cae87420dfaca96876d787b88a46f04f9d61eefd9c8abb" gracePeriod=30 Nov 02 21:35:25.878297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.878273 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:35:25.878463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.878444 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" containerID="cri-o://587d85bc4f4297f7fef0a229dd48273cce4b5201392d2751060a9fb4271a12bd" gracePeriod=30 Nov 02 21:35:25.925900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.925875 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:35:25.926089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:25.926070 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" containerID="cri-o://3219815b3d66acefddc2dd296c9fd983636a6274310c4bd318c344a40b163727" gracePeriod=30 Nov 02 21:35:25.949907 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:25.949872 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:35:29.561543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:29.561507 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:29.561913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:29.561560 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:29.666693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:29.666656 5859 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:29.666846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:29.666721 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:31.164772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.164305 5859 generic.go:356] "Generic (PLEG): container finished" podID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerID="a67da46892b3e388f3cae87420dfaca96876d787b88a46f04f9d61eefd9c8abb" exitCode=0 Nov 02 21:35:31.164772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.164461 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" event={"ID":"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd","Type":"ContainerDied","Data":"a67da46892b3e388f3cae87420dfaca96876d787b88a46f04f9d61eefd9c8abb"} Nov 02 21:35:31.168551 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.168490 5859 generic.go:356] "Generic (PLEG): container finished" podID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerID="587d85bc4f4297f7fef0a229dd48273cce4b5201392d2751060a9fb4271a12bd" exitCode=0 Nov 02 21:35:31.168708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.168660 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" event={"ID":"c4ace614-33ba-4f58-b987-25e50dd83dd8","Type":"ContainerDied","Data":"587d85bc4f4297f7fef0a229dd48273cce4b5201392d2751060a9fb4271a12bd"} Nov 02 21:35:31.171894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.171863 5859 generic.go:356] "Generic (PLEG): container finished" podID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerID="3219815b3d66acefddc2dd296c9fd983636a6274310c4bd318c344a40b163727" exitCode=0 Nov 02 21:35:31.172037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.172019 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" event={"ID":"c3dd1bfb-1d46-49bb-a209-84f046e20f9c","Type":"ContainerDied","Data":"3219815b3d66acefddc2dd296c9fd983636a6274310c4bd318c344a40b163727"} Nov 02 21:35:31.173535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.173512 5859 generic.go:356] "Generic (PLEG): container finished" podID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerID="7fce3676056670c607412de9fe46a5dd6e0d9b5061991b85f063a58f7afc39f0" exitCode=0 Nov 02 21:35:31.173634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.173549 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" event={"ID":"669ae170-5bfa-4bbe-8037-5d510aca4f7f","Type":"ContainerDied","Data":"7fce3676056670c607412de9fe46a5dd6e0d9b5061991b85f063a58f7afc39f0"} Nov 02 21:35:31.202677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.202651 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.267932 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268019 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268084 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268144 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268184 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268253 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268314 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268365 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.268458 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\") pod \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\" (UID: \"669ae170-5bfa-4bbe-8037-5d510aca4f7f\") " Nov 02 21:35:31.269559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.269534 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.270137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.269839 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:31.271047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.270988 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.271047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.270924 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.271201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.271038 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.272878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.272851 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.274551 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.274530 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "kube-api-access-24kz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.275989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.275504 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.276862 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.276832 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo" pod "669ae170-5bfa-4bbe-8037-5d510aca4f7f" (UID: "669ae170-5bfa-4bbe-8037-5d510aca4f7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:31.287665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.287641 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:35:31.330277 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.330255 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:35:31.338013 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.337989 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:35:31.369724 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369695 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.369880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369787 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.369880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369849 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.369981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369881 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.369981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369902 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.369981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369936 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.370124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.369986 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.370124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370034 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370112 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370156 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.370264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370207 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.370264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370229 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.370264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370257 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.370443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370289 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.370443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370333 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370361 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.370443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370392 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370418 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370460 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370492 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370540 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370582 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370610 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\") pod \"c4ace614-33ba-4f58-b987-25e50dd83dd8\" (UID: \"c4ace614-33ba-4f58-b987-25e50dd83dd8\") " Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370629 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.370694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370684 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370718 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370750 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370772 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370795 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\") pod \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\" (UID: \"c3dd1bfb-1d46-49bb-a209-84f046e20f9c\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370814 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\") pod \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\" (UID: \"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd\") " Nov 02 21:35:31.371095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.370842 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371481 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371506 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371523 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371538 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371553 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371567 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371582 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371596 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371610 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.371630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371627 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kz5\" (UniqueName: \"kubernetes.io/projected/669ae170-5bfa-4bbe-8037-5d510aca4f7f-kube-api-access-24kz5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.372225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371641 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.372225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371658 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669ae170-5bfa-4bbe-8037-5d510aca4f7f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.372225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.371804 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:31.372377 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.372358 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.372554 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.372532 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.372853 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.372863 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.373201 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.373449 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.373525 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.374134 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.374036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.375678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.374663 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.376976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.376029 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.376976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.376362 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "kube-api-access-jcbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.377413 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.377366 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:31.377502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.377483 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "kube-api-access-2845j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.377624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.377605 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.377792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.377739 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.378220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.378176 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.378220 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.378201 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:31.378371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.378213 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:31.378425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.378402 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "kube-api-access-hgmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.379035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.379009 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy" pod "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" (UID: "fd7eab2d-d875-4f9f-9eed-958e19d4c4dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:31.379909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.379882 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token" pod "c4ace614-33ba-4f58-b987-25e50dd83dd8" (UID: "c4ace614-33ba-4f58-b987-25e50dd83dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.380070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.379918 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token" pod "c3dd1bfb-1d46-49bb-a209-84f046e20f9c" (UID: "c3dd1bfb-1d46-49bb-a209-84f046e20f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:31.472920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472869 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.472920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472891 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.472920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472902 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.472920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472911 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.472920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472920 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472928 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgmgh\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-kube-api-access-hgmgh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472936 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472944 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472966 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ace614-33ba-4f58-b987-25e50dd83dd8-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472975 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472984 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.472992 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473002 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473010 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473018 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473026 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473034 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473043 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbg8\" (UniqueName: \"kubernetes.io/projected/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-kube-api-access-jcbg8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473051 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473059 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2845j\" (UniqueName: \"kubernetes.io/projected/c4ace614-33ba-4f58-b987-25e50dd83dd8-kube-api-access-2845j\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473067 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473074 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3dd1bfb-1d46-49bb-a209-84f046e20f9c-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473082 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:31.473174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:31.473091 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ace614-33ba-4f58-b987-25e50dd83dd8-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:32.181248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.181175 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" Nov 02 21:35:32.181248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.181214 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j" event={"ID":"669ae170-5bfa-4bbe-8037-5d510aca4f7f","Type":"ContainerDied","Data":"53e2a503aaec9aaa52397a0a161931a407091e13ece5f85b1a28e0d4bceb913d"} Nov 02 21:35:32.181767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.181282 5859 scope.go:122] "RemoveContainer" containerID="7fce3676056670c607412de9fe46a5dd6e0d9b5061991b85f063a58f7afc39f0" Nov 02 21:35:32.182711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.182682 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" Nov 02 21:35:32.182838 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.182764 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph" event={"ID":"fd7eab2d-d875-4f9f-9eed-958e19d4c4dd","Type":"ContainerDied","Data":"1d056d248da57dac6badf36160cf03263e2a97a6c3c971e855db1d76f752051c"} Nov 02 21:35:32.184339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.184319 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" Nov 02 21:35:32.184339 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.184319 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x" event={"ID":"c4ace614-33ba-4f58-b987-25e50dd83dd8","Type":"ContainerDied","Data":"eeeba8b0bf20dc48aa8ad3eea1ea43a435277c2375a907c5408a11b9701dea5b"} Nov 02 21:35:32.185885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.185864 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" event={"ID":"c3dd1bfb-1d46-49bb-a209-84f046e20f9c","Type":"ContainerDied","Data":"bd4f8010438b7528956614815a30471713c8368902daa851626ae9fe7db8568c"} Nov 02 21:35:32.186010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.185889 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9" Nov 02 21:35:32.205732 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.205523 5859 scope.go:122] "RemoveContainer" containerID="a67da46892b3e388f3cae87420dfaca96876d787b88a46f04f9d61eefd9c8abb" Nov 02 21:35:32.216029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.216011 5859 scope.go:122] "RemoveContainer" containerID="587d85bc4f4297f7fef0a229dd48273cce4b5201392d2751060a9fb4271a12bd" Nov 02 21:35:32.224438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.224420 5859 scope.go:122] "RemoveContainer" containerID="3219815b3d66acefddc2dd296c9fd983636a6274310c4bd318c344a40b163727" Nov 02 21:35:32.348174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.348144 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:35:32.398255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.398228 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-ktj7x"] Nov 02 21:35:32.463652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.463568 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:35:32.467476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.467446 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-fzkb9"] Nov 02 21:35:32.572359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.572326 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" path="/var/lib/kubelet/pods/c3dd1bfb-1d46-49bb-a209-84f046e20f9c/volumes" Nov 02 21:35:32.572833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.572819 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" path="/var/lib/kubelet/pods/c4ace614-33ba-4f58-b987-25e50dd83dd8/volumes" Nov 02 21:35:32.578240 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.578218 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:35:32.583405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.583373 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-2w52j"] Nov 02 21:35:32.693159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.693132 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:35:32.695944 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:32.695920 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-zdtph"] Nov 02 21:35:34.570914 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.570883 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" path="/var/lib/kubelet/pods/669ae170-5bfa-4bbe-8037-5d510aca4f7f/volumes" Nov 02 21:35:34.571422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.571407 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" path="/var/lib/kubelet/pods/fd7eab2d-d875-4f9f-9eed-958e19d4c4dd/volumes" Nov 02 21:35:34.723448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.723412 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:35:34.724083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724060 5859 cpu_manager.go:414] "containerMap: removing container" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" Nov 02 21:35:34.724083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724082 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724113 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724121 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724135 5859 cpu_manager.go:414] "containerMap: removing container" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724142 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724161 5859 cpu_manager.go:414] "containerMap: removing container" podUID="0ccf7da5-873e-4086-bf13-d4818a5f535b" containerName="payload-processing" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724168 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccf7da5-873e-4086-bf13-d4818a5f535b" containerName="payload-processing" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724179 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c01ec63f-6f17-49fd-87fd-667173659e42" containerName="payload-pre-processing" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724186 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01ec63f-6f17-49fd-87fd-667173659e42" containerName="payload-pre-processing" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724200 5859 cpu_manager.go:414] "containerMap: removing container" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724207 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724226 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724233 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724244 5859 cpu_manager.go:414] "containerMap: removing container" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724252 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" Nov 02 21:35:34.724272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724276 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724283 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724296 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724302 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724314 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" containerName="payload-processing" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724322 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" containerName="payload-processing" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724481 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8164fb9-c683-42ec-956f-094cec96fc34" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724493 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724502 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccf7da5-873e-4086-bf13-d4818a5f535b" containerName="payload-processing" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724514 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01ec63f-6f17-49fd-87fd-667173659e42" containerName="payload-pre-processing" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724526 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="63501607-b967-47e3-8aef-6f8ac2b9cd8b" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724536 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12214cb-123b-42e1-be01-a2984b425fbe" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724544 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="669ae170-5bfa-4bbe-8037-5d510aca4f7f" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724557 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ace614-33ba-4f58-b987-25e50dd83dd8" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724567 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dd1bfb-1d46-49bb-a209-84f046e20f9c" containerName="istio-proxy" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724578 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc2f1ee-7639-4ecb-8325-dd7877c97595" containerName="payload-processing" Nov 02 21:35:34.724934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.724592 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7eab2d-d875-4f9f-9eed-958e19d4c4dd" containerName="istio-proxy" Nov 02 21:35:34.730128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.730108 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.791739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.791707 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:35:34.815817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.815548 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.815817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.815597 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.815817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.815642 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.815817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.815796 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.816157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.815923 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.816157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.816062 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.816248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.816157 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4zz\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.816248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.816221 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.816348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.816265 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.832529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.832458 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6606aa-openshift-default-dockercfg-mqvgs\"" Nov 02 21:35:34.917763 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.917726 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.917973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.917804 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.917973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.917861 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.917973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.917936 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918033 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918191 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918274 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4zz\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918332 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918344 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918410 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918791 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918767 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918792 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918835 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.918931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.918868 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.920510 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.920488 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.920635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.920619 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.975916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.975879 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:34.976076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:34.975982 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4zz\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\") pod \"e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:35.043703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.043671 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:35.491368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.491342 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:35:35.492278 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:35:35.492252 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3da1d4_bc4c_48fb_b1ea_49bb783c0635.slice/crio-f6e26de9da1746f6e0d5382a795ff3f0b0e53e1ac9ab9239eaca6a2a568d5080 WatchSource:0}: Error finding container f6e26de9da1746f6e0d5382a795ff3f0b0e53e1ac9ab9239eaca6a2a568d5080: Status 404 returned error can't find the container with id f6e26de9da1746f6e0d5382a795ff3f0b0e53e1ac9ab9239eaca6a2a568d5080 Nov 02 21:35:35.494810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.494776 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.494927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.494853 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.494927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.494898 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.748662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.748567 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:35:35.749142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:35.748803 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" podUID="ab76a576-ad6f-43c9-9cda-84646104a01f" containerName="maas-api" containerID="cri-o://86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec" gracePeriod=30 Nov 02 21:35:36.033401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.033368 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:35:36.137712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.136825 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") pod \"ab76a576-ad6f-43c9-9cda-84646104a01f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " Nov 02 21:35:36.137712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.137007 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\") pod \"ab76a576-ad6f-43c9-9cda-84646104a01f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " Nov 02 21:35:36.137712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.137137 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\") pod \"ab76a576-ad6f-43c9-9cda-84646104a01f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " Nov 02 21:35:36.137712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.137230 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\" (UniqueName: \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\") pod \"ab76a576-ad6f-43c9-9cda-84646104a01f\" (UID: \"ab76a576-ad6f-43c9-9cda-84646104a01f\") " Nov 02 21:35:36.137712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.137673 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca" pod "ab76a576-ad6f-43c9-9cda-84646104a01f" (UID: "ab76a576-ad6f-43c9-9cda-84646104a01f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:36.138141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.137966 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab76a576-ad6f-43c9-9cda-84646104a01f-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:36.139391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.139361 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls" pod "ab76a576-ad6f-43c9-9cda-84646104a01f" (UID: "ab76a576-ad6f-43c9-9cda-84646104a01f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:36.139391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.139384 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs" pod "ab76a576-ad6f-43c9-9cda-84646104a01f" (UID: "ab76a576-ad6f-43c9-9cda-84646104a01f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:36.139894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.139867 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw" pod "ab76a576-ad6f-43c9-9cda-84646104a01f" (UID: "ab76a576-ad6f-43c9-9cda-84646104a01f"). InnerVolumeSpecName "kube-api-access-h57zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:36.217391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.217351 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerStarted","Data":"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453"} Nov 02 21:35:36.217516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.217419 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerStarted","Data":"f6e26de9da1746f6e0d5382a795ff3f0b0e53e1ac9ab9239eaca6a2a568d5080"} Nov 02 21:35:36.218725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.218702 5859 generic.go:356] "Generic (PLEG): container finished" podID="ab76a576-ad6f-43c9-9cda-84646104a01f" containerID="86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec" exitCode=0 Nov 02 21:35:36.218795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.218736 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" Nov 02 21:35:36.218795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.218784 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" event={"ID":"ab76a576-ad6f-43c9-9cda-84646104a01f","Type":"ContainerDied","Data":"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec"} Nov 02 21:35:36.218868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.218828 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f" event={"ID":"ab76a576-ad6f-43c9-9cda-84646104a01f","Type":"ContainerDied","Data":"c11dc8046d2d131db68a07ac1864a8d68dc81c1b4e2c23fabd72319a666d6300"} Nov 02 21:35:36.218868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.218849 5859 scope.go:122] "RemoveContainer" containerID="86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec" Nov 02 21:35:36.238631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.238611 5859 scope.go:122] "RemoveContainer" containerID="86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec" Nov 02 21:35:36.238776 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.238757 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h57zw\" (UniqueName: \"kubernetes.io/projected/ab76a576-ad6f-43c9-9cda-84646104a01f-kube-api-access-h57zw\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:36.238826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.238784 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:36.238826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.238798 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab76a576-ad6f-43c9-9cda-84646104a01f-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:36.238926 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:36.238906 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec\": container with ID starting with 86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec not found: ID does not exist" containerID="86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec" Nov 02 21:35:36.238983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.238938 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec"} err="failed to get container status \"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec\": rpc error: code = NotFound desc = could not find container \"86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec\": container with ID starting with 86745e9ae00d5647bb18f3d55d9bc3c47b34f3945703640794c920649c1ca2ec not found: ID does not exist" Nov 02 21:35:36.338215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.338160 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podStartSLOduration=2.338146994 podStartE2EDuration="2.338146994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:36.335928419 +0000 UTC m=+1575.884178223" watchObservedRunningTime="2026-11-02 21:35:36.338146994 +0000 UTC m=+1575.886396833" Nov 02 21:35:36.461081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.461045 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:35:36.472239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.472211 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hz78f"] Nov 02 21:35:36.570867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:36.570836 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab76a576-ad6f-43c9-9cda-84646104a01f" path="/var/lib/kubelet/pods/ab76a576-ad6f-43c9-9cda-84646104a01f/volumes" Nov 02 21:35:37.043827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:37.043794 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:37.050189 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:37.050162 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:37.226706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:37.226684 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:37.227404 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:37.227384 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:35:44.274699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.274665 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:35:44.275192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.274874 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" containerID="cri-o://16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad" gracePeriod=30 Nov 02 21:35:44.325271 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.325238 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:35:44.325540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.325509 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" containerID="cri-o://62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb" gracePeriod=30 Nov 02 21:35:44.328508 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.328482 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:35:44.328712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.328678 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" containerID="cri-o://dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6" gracePeriod=30 Nov 02 21:35:44.376676 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.376642 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:35:44.376878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:44.376847 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" containerID="cri-o://4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624" gracePeriod=30 Nov 02 21:35:45.195043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.194908 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:35:45.195613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.195590 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ab76a576-ad6f-43c9-9cda-84646104a01f" containerName="maas-api" Nov 02 21:35:45.195613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.195611 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab76a576-ad6f-43c9-9cda-84646104a01f" containerName="maas-api" Nov 02 21:35:45.195820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.195760 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab76a576-ad6f-43c9-9cda-84646104a01f" containerName="maas-api" Nov 02 21:35:45.200150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.200098 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.252032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.252002 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:35:45.253226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.253191 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6606aa\"" Nov 02 21:35:45.291459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.291432 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.291777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.291559 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmdn\" (UniqueName: \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.353915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.353882 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:35:45.358282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.358262 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.393200 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.393156 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.393383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.393215 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.393383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.393315 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmdn\" (UniqueName: \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.393527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.393422 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlxds\" (UniqueName: \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.394480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.394439 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.422845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.422809 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:35:45.423213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.423170 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6606aa-dockercfg-9qmnj\"" Nov 02 21:35:45.457018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.455822 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmdn\" (UniqueName: \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\") pod \"payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.495089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.495055 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlxds\" (UniqueName: \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.495252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.495216 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.495907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.495880 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.517737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.517712 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:45.552076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.552052 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlxds\" (UniqueName: \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\") pod \"payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.695032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.695002 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:45.899945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:45.899922 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:35:45.901573 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:35:45.901544 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2185fe8_4ec1_4ca6_ad91_907ab685c4e4.slice/crio-30e776aa63b9862984420d1415d13900714df9decad88660383b8583963aba74 WatchSource:0}: Error finding container 30e776aa63b9862984420d1415d13900714df9decad88660383b8583963aba74: Status 404 returned error can't find the container with id 30e776aa63b9862984420d1415d13900714df9decad88660383b8583963aba74 Nov 02 21:35:46.073188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.073161 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:35:46.074195 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:35:46.074168 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19119e0c_3768_4860_83fd_35addee2a3e3.slice/crio-1a513125c8f1c949020b45b22a1841b759f54cce43db21d219dfda5575626dd2 WatchSource:0}: Error finding container 1a513125c8f1c949020b45b22a1841b759f54cce43db21d219dfda5575626dd2: Status 404 returned error can't find the container with id 1a513125c8f1c949020b45b22a1841b759f54cce43db21d219dfda5575626dd2 Nov 02 21:35:46.292799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.292719 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" event={"ID":"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4","Type":"ContainerStarted","Data":"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d"} Nov 02 21:35:46.292799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.292791 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:46.293352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.292816 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" event={"ID":"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4","Type":"ContainerStarted","Data":"30e776aa63b9862984420d1415d13900714df9decad88660383b8583963aba74"} Nov 02 21:35:46.294597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.294573 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" event={"ID":"19119e0c-3768-4860-83fd-35addee2a3e3","Type":"ContainerStarted","Data":"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8"} Nov 02 21:35:46.294705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.294613 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" event={"ID":"19119e0c-3768-4860-83fd-35addee2a3e3","Type":"ContainerStarted","Data":"1a513125c8f1c949020b45b22a1841b759f54cce43db21d219dfda5575626dd2"} Nov 02 21:35:46.294705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.294644 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:35:46.411396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:46.411347 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" podStartSLOduration=1.4113337910000001 podStartE2EDuration="1.411333791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:46.408664116 +0000 UTC m=+1585.956913911" watchObservedRunningTime="2026-11-02 21:35:46.411333791 +0000 UTC m=+1585.959583571" Nov 02 21:35:49.619041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.619016 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:35:49.743849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.743827 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:35:49.772193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.772169 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:35:49.785883 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.785855 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.786386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.785908 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.786386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.786273 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:49.786636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.786614 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.787222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787202 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787321 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787284 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787325 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787360 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787410 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787428 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.787490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787428 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.787490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787472 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\") pod \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\" (UID: \"1ed64869-14d9-4a52-ac1e-d78c1f79d95c\") " Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.787843 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.788031 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.788050 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.788062 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.788075 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.788428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.788305 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.792997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.790546 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.792997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.791125 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.792997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.791587 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "kube-api-access-x9mlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.794869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.794845 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo" pod "1ed64869-14d9-4a52-ac1e-d78c1f79d95c" (UID: "1ed64869-14d9-4a52-ac1e-d78c1f79d95c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:49.797620 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.797596 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:35:49.825662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.825594 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" podStartSLOduration=4.825579919 podStartE2EDuration="4.825579919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:46.527195041 +0000 UTC m=+1586.075444826" watchObservedRunningTime="2026-11-02 21:35:49.825579919 +0000 UTC m=+1589.373829701" Nov 02 21:35:49.889495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889042 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889689 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889773 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889855 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889897 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.889943 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890000 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890039 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890095 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890129 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890171 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890195 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890242 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890277 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890344 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.890693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.890450 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892372 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892427 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892472 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892501 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892539 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892569 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892627 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892676 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892722 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892757 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\") pod \"012bc30b-1291-4f15-be9a-fcfcd851491f\" (UID: \"012bc30b-1291-4f15-be9a-fcfcd851491f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892783 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\") pod \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\" (UID: \"7276909f-cbc2-42a5-bc13-c844cfcd31d8\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.892830 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\") pod \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\" (UID: \"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f\") " Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893600 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893620 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893627 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893634 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893671 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.893691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893691 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.894592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893707 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mlb\" (UniqueName: \"kubernetes.io/projected/1ed64869-14d9-4a52-ac1e-d78c1f79d95c-kube-api-access-x9mlb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.894592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.893930 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.894592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.894385 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:49.894745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.894704 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:49.895353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.895015 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.896546 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.896735 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.896866 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.896934 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "kube-api-access-gb9gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.897181 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.897356 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.897632 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.897556 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:49.898088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.897824 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.898088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.898023 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.898203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.898188 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.898643 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.898359 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.899464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.899355 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.899865 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.899840 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:49.900327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.900269 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.900791 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.900764 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.901540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.901468 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.901761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.901743 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw" pod "012bc30b-1291-4f15-be9a-fcfcd851491f" (UID: "012bc30b-1291-4f15-be9a-fcfcd851491f"). InnerVolumeSpecName "kube-api-access-pt7jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.903594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.903570 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "kube-api-access-gtknl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:49.903680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.903579 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:49.904546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.904509 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy" pod "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" (UID: "0f973eaa-27f9-4197-8c9d-1e6b0aafa86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:49.906075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.906042 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo" pod "7276909f-cbc2-42a5-bc13-c844cfcd31d8" (UID: "7276909f-cbc2-42a5-bc13-c844cfcd31d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:49.961439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.961405 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:35:49.962061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962046 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962062 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962078 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962083 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962112 5859 cpu_manager.go:414] "containerMap: removing container" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962118 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962133 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.962137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962137 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962159 5859 cpu_manager.go:414] "containerMap: removing container" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962164 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962252 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962263 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962275 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962283 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerName="istio-proxy" Nov 02 21:35:49.962334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.962290 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerName="istio-proxy" Nov 02 21:35:49.967030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.967015 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:49.994746 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994724 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994751 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994766 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012bc30b-1291-4f15-be9a-fcfcd851491f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994780 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994793 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994806 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.994832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994818 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994831 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994847 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994860 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994872 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994886 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9gf\" (UniqueName: \"kubernetes.io/projected/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-kube-api-access-gb9gf\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994898 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994911 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtknl\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-kube-api-access-gtknl\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994923 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994935 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994979 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.994994 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995007 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995020 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995033 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995045 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995058 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012bc30b-1291-4f15-be9a-fcfcd851491f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995071 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7276909f-cbc2-42a5-bc13-c844cfcd31d8-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995084 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7jw\" (UniqueName: \"kubernetes.io/projected/012bc30b-1291-4f15-be9a-fcfcd851491f-kube-api-access-pt7jw\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:49.995135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:49.995098 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7276909f-cbc2-42a5-bc13-c844cfcd31d8-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:50.020188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.020156 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-70efbbab\"" Nov 02 21:35:50.020434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.020412 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:35:50.096186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.096120 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcjw\" (UniqueName: \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.096395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.096261 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.125346 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.125321 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:35:50.129894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.129874 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.184711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.182853 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-70efbbab-dockercfg-6jztw\"" Nov 02 21:35:50.184711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.184702 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:35:50.211599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.211556 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcjw\" (UniqueName: \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.211785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.211677 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.215111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.215079 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.273340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.273308 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcjw\" (UniqueName: \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\") pod \"payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.280777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.280754 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:50.312701 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.312673 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.312862 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.312841 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzm4\" (UniqueName: \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.324980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.324936 5859 generic.go:356] "Generic (PLEG): container finished" podID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" containerID="16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad" exitCode=0 Nov 02 21:35:50.325113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.324986 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerDied","Data":"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad"} Nov 02 21:35:50.325113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.325015 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" Nov 02 21:35:50.325113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.325046 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9" event={"ID":"1ed64869-14d9-4a52-ac1e-d78c1f79d95c","Type":"ContainerDied","Data":"6a847a5e27e7da42823c03c1dfa52f2ed845a2c2fdf903c9e35b2d20ec9554da"} Nov 02 21:35:50.325113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.325072 5859 scope.go:122] "RemoveContainer" containerID="16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad" Nov 02 21:35:50.328034 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.328013 5859 generic.go:356] "Generic (PLEG): container finished" podID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" containerID="62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb" exitCode=0 Nov 02 21:35:50.328156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.328114 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" event={"ID":"7276909f-cbc2-42a5-bc13-c844cfcd31d8","Type":"ContainerDied","Data":"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb"} Nov 02 21:35:50.328156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.328146 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" event={"ID":"7276909f-cbc2-42a5-bc13-c844cfcd31d8","Type":"ContainerDied","Data":"0f33c7cf6703c23c335bdefe0dbdb9e8fe22a388e37649e192915714eacb62b0"} Nov 02 21:35:50.328282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.328209 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2" Nov 02 21:35:50.330943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.330924 5859 generic.go:356] "Generic (PLEG): container finished" podID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" containerID="dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6" exitCode=0 Nov 02 21:35:50.331076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.331055 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" event={"ID":"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f","Type":"ContainerDied","Data":"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6"} Nov 02 21:35:50.331174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.331096 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" event={"ID":"0f973eaa-27f9-4197-8c9d-1e6b0aafa86f","Type":"ContainerDied","Data":"8939db874d6065554b97251ab00f969fad51ad0f41eac43eabc9aa67aea57818"} Nov 02 21:35:50.331245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.331172 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp" Nov 02 21:35:50.334212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.334088 5859 generic.go:356] "Generic (PLEG): container finished" podID="012bc30b-1291-4f15-be9a-fcfcd851491f" containerID="4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624" exitCode=0 Nov 02 21:35:50.334212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.334141 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" event={"ID":"012bc30b-1291-4f15-be9a-fcfcd851491f","Type":"ContainerDied","Data":"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624"} Nov 02 21:35:50.334212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.334172 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" Nov 02 21:35:50.334212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.334181 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf" event={"ID":"012bc30b-1291-4f15-be9a-fcfcd851491f","Type":"ContainerDied","Data":"68ef6a61b06c76b6140ef4653ac1f71edc9e96f1565d124ebe1f9403427a23db"} Nov 02 21:35:50.382721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.382697 5859 scope.go:122] "RemoveContainer" containerID="ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad" Nov 02 21:35:50.408437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.408406 5859 scope.go:122] "RemoveContainer" containerID="16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad" Nov 02 21:35:50.408869 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:50.408837 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad\": container with ID starting with 16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad not found: ID does not exist" containerID="16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad" Nov 02 21:35:50.409076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.408882 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad"} err="failed to get container status \"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad\": rpc error: code = NotFound desc = could not find container \"16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad\": container with ID starting with 16d0f8d66bc2dbef2474a88a063a4f0afe92fadc43d133342ae59d441b5635ad not found: ID does not exist" Nov 02 21:35:50.409076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.408908 5859 scope.go:122] "RemoveContainer" containerID="ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad" Nov 02 21:35:50.410775 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:50.410753 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad\": container with ID starting with ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad not found: ID does not exist" containerID="ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad" Nov 02 21:35:50.410859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.410780 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad"} err="failed to get container status \"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad\": rpc error: code = NotFound desc = could not find container \"ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad\": container with ID starting with ea59b2a08ac497ef76323e46bb6ebea1bb2c36d6a0c7fde19de75a8ccb20dbad not found: ID does not exist" Nov 02 21:35:50.410859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.410797 5859 scope.go:122] "RemoveContainer" containerID="62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb" Nov 02 21:35:50.415090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.414840 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.415090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.414994 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzm4\" (UniqueName: \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.415617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.415598 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.424759 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.424728 5859 scope.go:122] "RemoveContainer" containerID="62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb" Nov 02 21:35:50.425166 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:50.425137 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb\": container with ID starting with 62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb not found: ID does not exist" containerID="62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb" Nov 02 21:35:50.425263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.425175 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb"} err="failed to get container status \"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb\": rpc error: code = NotFound desc = could not find container \"62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb\": container with ID starting with 62970010509a9645b9b603f0bedd657f075f9740b99a8ec2e570bac80e3387cb not found: ID does not exist" Nov 02 21:35:50.425263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.425218 5859 scope.go:122] "RemoveContainer" containerID="dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6" Nov 02 21:35:50.437796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.437673 5859 scope.go:122] "RemoveContainer" containerID="dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6" Nov 02 21:35:50.438021 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:50.437999 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6\": container with ID starting with dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6 not found: ID does not exist" containerID="dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6" Nov 02 21:35:50.438106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.438030 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6"} err="failed to get container status \"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6\": rpc error: code = NotFound desc = could not find container \"dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6\": container with ID starting with dc679e2e6af2f16675267b9d6f02538bf9e93d2b4e48f4b59defba07be4fa4a6 not found: ID does not exist" Nov 02 21:35:50.438106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.438051 5859 scope.go:122] "RemoveContainer" containerID="4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624" Nov 02 21:35:50.449131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.449112 5859 scope.go:122] "RemoveContainer" containerID="4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624" Nov 02 21:35:50.449407 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:50.449385 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624\": container with ID starting with 4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624 not found: ID does not exist" containerID="4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624" Nov 02 21:35:50.449479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.449412 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624"} err="failed to get container status \"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624\": rpc error: code = NotFound desc = could not find container \"4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624\": container with ID starting with 4a95bff601c27c8e7cd02ad9a230999fea4f9ca92814914968c077e623073624 not found: ID does not exist" Nov 02 21:35:50.472096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.472078 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzm4\" (UniqueName: \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\") pod \"payload-processing-e2e-mt-70efbbab-79796576df-bl5pq\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.675764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.675741 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:35:50.677302 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:35:50.677271 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269174e2_caa4_4c25_a58e_30dbd9eed84f.slice/crio-6528bca5c5fc2be50f6b5be35d03c82ffb04a46f147dc5f7becea31d430439ea WatchSource:0}: Error finding container 6528bca5c5fc2be50f6b5be35d03c82ffb04a46f147dc5f7becea31d430439ea: Status 404 returned error can't find the container with id 6528bca5c5fc2be50f6b5be35d03c82ffb04a46f147dc5f7becea31d430439ea Nov 02 21:35:50.688518 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.688238 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:35:50.691214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.691194 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zp6sp"] Nov 02 21:35:50.751526 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.751503 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:50.802991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.800498 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:35:50.812376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.812352 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-zlgr2"] Nov 02 21:35:50.915872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.915839 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:35:50.918488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:50.918466 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-6nrh9"] Nov 02 21:35:51.029444 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.029374 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:35:51.033821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.033799 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-5p2tf"] Nov 02 21:35:51.124281 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.124258 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:35:51.125719 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:35:51.125696 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f486811_c89f_4781_b561_3bc8972e759f.slice/crio-3d5bcc4c4cbdced49cd223bd8abcabc734a29ee3725914d49507c8b06d26be7f WatchSource:0}: Error finding container 3d5bcc4c4cbdced49cd223bd8abcabc734a29ee3725914d49507c8b06d26be7f: Status 404 returned error can't find the container with id 3d5bcc4c4cbdced49cd223bd8abcabc734a29ee3725914d49507c8b06d26be7f Nov 02 21:35:51.344901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.344866 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" event={"ID":"269174e2-caa4-4c25-a58e-30dbd9eed84f","Type":"ContainerStarted","Data":"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848"} Nov 02 21:35:51.345081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.344924 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" event={"ID":"269174e2-caa4-4c25-a58e-30dbd9eed84f","Type":"ContainerStarted","Data":"6528bca5c5fc2be50f6b5be35d03c82ffb04a46f147dc5f7becea31d430439ea"} Nov 02 21:35:51.345133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.345100 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:35:51.346414 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.346390 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" event={"ID":"2f486811-c89f-4781-b561-3bc8972e759f","Type":"ContainerStarted","Data":"80e430bef30547fea3df06a003fe4c38ec00f92aff75c3584f88b63857ebd2e9"} Nov 02 21:35:51.346537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.346425 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" event={"ID":"2f486811-c89f-4781-b561-3bc8972e759f","Type":"ContainerStarted","Data":"3d5bcc4c4cbdced49cd223bd8abcabc734a29ee3725914d49507c8b06d26be7f"} Nov 02 21:35:51.346537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.346471 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:35:51.458559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.458517 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" podStartSLOduration=2.458504455 podStartE2EDuration="2.458504455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:51.458019055 +0000 UTC m=+1591.006268848" watchObservedRunningTime="2026-11-02 21:35:51.458504455 +0000 UTC m=+1591.006754269" Nov 02 21:35:51.573831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:51.573778 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" podStartSLOduration=1.5737620620000001 podStartE2EDuration="1.573762062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:51.571611797 +0000 UTC m=+1591.119861604" watchObservedRunningTime="2026-11-02 21:35:51.573762062 +0000 UTC m=+1591.122011881" Nov 02 21:35:52.571657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:52.571627 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012bc30b-1291-4f15-be9a-fcfcd851491f" path="/var/lib/kubelet/pods/012bc30b-1291-4f15-be9a-fcfcd851491f/volumes" Nov 02 21:35:52.572105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:52.572090 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f973eaa-27f9-4197-8c9d-1e6b0aafa86f" path="/var/lib/kubelet/pods/0f973eaa-27f9-4197-8c9d-1e6b0aafa86f/volumes" Nov 02 21:35:52.572521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:52.572508 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed64869-14d9-4a52-ac1e-d78c1f79d95c" path="/var/lib/kubelet/pods/1ed64869-14d9-4a52-ac1e-d78c1f79d95c/volumes" Nov 02 21:35:52.573262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:52.573246 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7276909f-cbc2-42a5-bc13-c844cfcd31d8" path="/var/lib/kubelet/pods/7276909f-cbc2-42a5-bc13-c844cfcd31d8/volumes" Nov 02 21:35:55.946427 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:35:55.946385 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:35:57.304792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:57.304763 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:35:57.305231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:35:57.304944 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:36:02.356906 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:02.356874 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:36:02.357391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:02.356959 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:36:19.234873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.234837 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:36:19.239266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.239244 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259066 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259125 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvwrn\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259167 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259199 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259268 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259319 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259366 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259440 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.259722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.259517 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.314873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.314841 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:36:19.361280 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361236 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361300 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvwrn\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361340 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361369 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361439 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361724 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361776 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361794 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.361924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361878 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.362001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.361983 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.362262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.362239 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.362588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.362561 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-54e844-openshift-default-dockercfg-lscvs\"" Nov 02 21:36:19.362681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.362592 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.362681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.362573 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.362777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.362704 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.364790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.364763 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.364790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.364777 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.419471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.419433 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.419683 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.419571 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvwrn\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\") pod \"e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.556978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.556868 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:19.992491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:19.992453 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:36:20.031813 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:36:20.031773 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58597b2e_a889_49dd_9f2c_b039f4251bbd.slice/crio-f988206c4135e0d3a7e329ef198dc0cb03d2fa293dd0473631c862e376368819 WatchSource:0}: Error finding container f988206c4135e0d3a7e329ef198dc0cb03d2fa293dd0473631c862e376368819: Status 404 returned error can't find the container with id f988206c4135e0d3a7e329ef198dc0cb03d2fa293dd0473631c862e376368819 Nov 02 21:36:20.034241 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.034207 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:20.034347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.034277 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:20.034347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.034306 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:20.542347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.542310 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerStarted","Data":"21d1d32b18611d5ae53df600c9199c5639938bc07cabb58cbcd0f715442465a0"} Nov 02 21:36:20.542796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.542380 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerStarted","Data":"f988206c4135e0d3a7e329ef198dc0cb03d2fa293dd0473631c862e376368819"} Nov 02 21:36:20.557747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.557715 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:20.661581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:20.661518 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podStartSLOduration=1.661502417 podStartE2EDuration="1.661502417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:20.658773359 +0000 UTC m=+1620.207023167" watchObservedRunningTime="2026-11-02 21:36:20.661502417 +0000 UTC m=+1620.209752198" Nov 02 21:36:21.288584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.288552 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:36:21.294325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.294303 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.346782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.346697 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-70efbbab-serving-cert\"" Nov 02 21:36:21.347497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.347469 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:36:21.388428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.388393 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbwc\" (UniqueName: \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.388428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.388426 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.388640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.388540 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.388640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.388605 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.490376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.490341 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.490558 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.490452 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.490614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.490585 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbwc\" (UniqueName: \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.490614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.490606 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.490795 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:21.490777 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-serving-cert: secret "maas-api-e2e-mt-70efbbab-serving-cert" not found Nov 02 21:36:21.490871 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:21.490860 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls podName:cfe05938-b807-48e3-91bd-3b19f4850aad nodeName:}" failed. No retries permitted until 2026-11-02 21:36:21.990839883 +0000 UTC m=+1621.539089656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls") pod "maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" (UID: "cfe05938-b807-48e3-91bd-3b19f4850aad") : secret "maas-api-e2e-mt-70efbbab-serving-cert" not found Nov 02 21:36:21.491253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.491232 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.492854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.492834 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.548621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.548586 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbwc\" (UniqueName: \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:21.569078 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.569047 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:21.997644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:21.997607 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:22.000066 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.000044 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") pod \"maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:22.207536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.207504 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:22.559557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.559477 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:22.560563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.560544 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:36:22.583229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.583205 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:36:22.583812 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:36:22.583785 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe05938_b807_48e3_91bd_3b19f4850aad.slice/crio-baa72711cdbb66fc94439dc7bf0039e134dac5c2e8421edb2e08639cb57e9c0e WatchSource:0}: Error finding container baa72711cdbb66fc94439dc7bf0039e134dac5c2e8421edb2e08639cb57e9c0e: Status 404 returned error can't find the container with id baa72711cdbb66fc94439dc7bf0039e134dac5c2e8421edb2e08639cb57e9c0e Nov 02 21:36:22.585253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:22.585232 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:36:23.569322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:23.569283 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" event={"ID":"cfe05938-b807-48e3-91bd-3b19f4850aad","Type":"ContainerStarted","Data":"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be"} Nov 02 21:36:23.569726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:23.569349 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" event={"ID":"cfe05938-b807-48e3-91bd-3b19f4850aad","Type":"ContainerStarted","Data":"baa72711cdbb66fc94439dc7bf0039e134dac5c2e8421edb2e08639cb57e9c0e"} Nov 02 21:36:23.716721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:23.716653 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" podStartSLOduration=1.825850162 podStartE2EDuration="2.716634023s" totalImagesPullingTime="890.783861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:23.714553381 +0000 UTC m=+1623.262803162" watchObservedRunningTime="2026-11-02 21:36:23.716634023 +0000 UTC m=+1623.264883804" Nov 02 21:36:24.590384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:24.590353 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:25.944300 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:25.944258 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:36:27.042245 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.042212 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:36:27.047150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.047119 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.103546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.103512 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:36:27.149272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.149237 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6606aa-serving-cert\"" Nov 02 21:36:27.170352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.170308 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.170539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.170374 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.170539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.170468 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.170663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.170639 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q96\" (UniqueName: \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.272094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.272060 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.272264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.272107 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.272264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.272176 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.272334 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:27.272277 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-serving-cert: secret "maas-api-e2e-api-b-6606aa-serving-cert" not found Nov 02 21:36:27.272334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.272287 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q96\" (UniqueName: \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.272401 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:27.272355 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls podName:c91e93eb-2f0e-4021-ad85-5e7c0018be87 nodeName:}" failed. No retries permitted until 2026-11-02 21:36:27.772335281 +0000 UTC m=+1627.320585057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls") pod "maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" (UID: "c91e93eb-2f0e-4021-ad85-5e7c0018be87") : secret "maas-api-e2e-api-b-6606aa-serving-cert" not found Nov 02 21:36:27.272782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.272763 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.274617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.274602 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.329825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.329797 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q96\" (UniqueName: \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.778440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.778352 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.780928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.780904 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") pod \"maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:27.964030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:27.963996 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:28.339208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:28.339179 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:36:28.339594 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:36:28.339574 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91e93eb_2f0e_4021_ad85_5e7c0018be87.slice/crio-6bb466e23de7ca2303922b61e348d673ae8a1d7d373da242495488fa23a3cf14 WatchSource:0}: Error finding container 6bb466e23de7ca2303922b61e348d673ae8a1d7d373da242495488fa23a3cf14: Status 404 returned error can't find the container with id 6bb466e23de7ca2303922b61e348d673ae8a1d7d373da242495488fa23a3cf14 Nov 02 21:36:28.618089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:28.618002 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" event={"ID":"c91e93eb-2f0e-4021-ad85-5e7c0018be87","Type":"ContainerStarted","Data":"6bb466e23de7ca2303922b61e348d673ae8a1d7d373da242495488fa23a3cf14"} Nov 02 21:36:29.626072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:29.626041 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" event={"ID":"c91e93eb-2f0e-4021-ad85-5e7c0018be87","Type":"ContainerStarted","Data":"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba"} Nov 02 21:36:29.626502 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:29.626180 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:29.741787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:29.741604 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" podStartSLOduration=2.69756505 podStartE2EDuration="3.741587029s" totalImagesPullingTime="1.044021979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:29.741304033 +0000 UTC m=+1629.289553832" watchObservedRunningTime="2026-11-02 21:36:29.741587029 +0000 UTC m=+1629.289836811" Nov 02 21:36:30.603715 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:30.603684 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:36:32.091896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.091819 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:36:32.092278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.092023 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" podUID="da006833-3684-4f85-bc40-d6ae795c5bc1" containerName="maas-api" containerID="cri-o://2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6" gracePeriod=30 Nov 02 21:36:32.376192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.376166 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:36:32.438229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438195 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\") pod \"da006833-3684-4f85-bc40-d6ae795c5bc1\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " Nov 02 21:36:32.438429 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438324 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") pod \"da006833-3684-4f85-bc40-d6ae795c5bc1\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " Nov 02 21:36:32.438429 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438360 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\") pod \"da006833-3684-4f85-bc40-d6ae795c5bc1\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " Nov 02 21:36:32.438429 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438388 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\" (UniqueName: \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\") pod \"da006833-3684-4f85-bc40-d6ae795c5bc1\" (UID: \"da006833-3684-4f85-bc40-d6ae795c5bc1\") " Nov 02 21:36:32.438649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438613 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca" pod "da006833-3684-4f85-bc40-d6ae795c5bc1" (UID: "da006833-3684-4f85-bc40-d6ae795c5bc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:32.438978 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.438926 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da006833-3684-4f85-bc40-d6ae795c5bc1-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:32.440634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.440606 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq" pod "da006833-3684-4f85-bc40-d6ae795c5bc1" (UID: "da006833-3684-4f85-bc40-d6ae795c5bc1"). InnerVolumeSpecName "kube-api-access-2xlsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:32.440875 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.440857 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls" pod "da006833-3684-4f85-bc40-d6ae795c5bc1" (UID: "da006833-3684-4f85-bc40-d6ae795c5bc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:32.440943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.440893 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs" pod "da006833-3684-4f85-bc40-d6ae795c5bc1" (UID: "da006833-3684-4f85-bc40-d6ae795c5bc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:32.540840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.540802 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:32.540840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.540836 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da006833-3684-4f85-bc40-d6ae795c5bc1-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:32.540840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.540848 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xlsq\" (UniqueName: \"kubernetes.io/projected/da006833-3684-4f85-bc40-d6ae795c5bc1-kube-api-access-2xlsq\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:32.647543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.647458 5859 generic.go:356] "Generic (PLEG): container finished" podID="da006833-3684-4f85-bc40-d6ae795c5bc1" containerID="2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6" exitCode=0 Nov 02 21:36:32.647543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.647505 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" Nov 02 21:36:32.647755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.647493 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" event={"ID":"da006833-3684-4f85-bc40-d6ae795c5bc1","Type":"ContainerDied","Data":"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6"} Nov 02 21:36:32.647755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.647615 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s" event={"ID":"da006833-3684-4f85-bc40-d6ae795c5bc1","Type":"ContainerDied","Data":"a8ff746a4dd7e27134684f3c785b82a93438fc5df69ae3fb79f7538fc7fab036"} Nov 02 21:36:32.647755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.647638 5859 scope.go:122] "RemoveContainer" containerID="2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6" Nov 02 21:36:32.657265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.657245 5859 scope.go:122] "RemoveContainer" containerID="2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6" Nov 02 21:36:32.657532 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:32.657511 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6\": container with ID starting with 2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6 not found: ID does not exist" containerID="2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6" Nov 02 21:36:32.657590 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.657542 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6"} err="failed to get container status \"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6\": rpc error: code = NotFound desc = could not find container \"2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6\": container with ID starting with 2afe5199a7214975a9133d1c96bac2777231df8f383cebaf35a018c5e8c97aa6 not found: ID does not exist" Nov 02 21:36:32.813295 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.813264 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:36:32.815961 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:32.815935 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bd96ef-69cc75c666-6v97s"] Nov 02 21:36:34.570822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:34.570790 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da006833-3684-4f85-bc40-d6ae795c5bc1" path="/var/lib/kubelet/pods/da006833-3684-4f85-bc40-d6ae795c5bc1/volumes" Nov 02 21:36:35.638186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:36:35.638155 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:36:55.964752 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:36:55.964716 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:37:13.939926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.939896 5859 generic.go:356] "Generic (PLEG): container finished" podID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerID="9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453" exitCode=0 Nov 02 21:37:13.940478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.939992 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerDied","Data":"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453"} Nov 02 21:37:13.940572 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.940555 5859 scope.go:122] "RemoveContainer" containerID="9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453" Nov 02 21:37:13.942159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.942129 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:13.942290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.942195 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:13.942290 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:13.942228 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:14.948653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:14.948619 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerStarted","Data":"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5"} Nov 02 21:37:15.043996 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:15.043962 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:37:15.044217 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:15.044196 5859 patch_prober.go:29] interesting pod/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:15.044269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:15.044251 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:16.044619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.044585 5859 patch_prober.go:29] interesting pod/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:16.045045 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.044636 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:16.094043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.094014 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:37:16.094598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.094585 5859 cpu_manager.go:414] "containerMap: removing container" podUID="da006833-3684-4f85-bc40-d6ae795c5bc1" containerName="maas-api" Nov 02 21:37:16.094645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.094598 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da006833-3684-4f85-bc40-d6ae795c5bc1" containerName="maas-api" Nov 02 21:37:16.094737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.094726 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="da006833-3684-4f85-bc40-d6ae795c5bc1" containerName="maas-api" Nov 02 21:37:16.119209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.119181 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.170585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170553 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.170706 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170690 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.170811 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170785 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98nb\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.170924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170816 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.170924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170866 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.171054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170929 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.171054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.170979 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.171119 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.171069 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.171176 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.171155 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.237521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.237477 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-70efbbab-openshift-default-dockercfg-m2wbl\"" Nov 02 21:37:16.242057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.242020 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:37:16.272855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.272816 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.272878 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98nb\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.272899 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.272919 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.272990 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273041 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273068 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273091 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273213 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273289 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273329 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273594 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273620 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.273695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.273631 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.275467 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.275441 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.275649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.275628 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.353439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.353375 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.354286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.354230 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98nb\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-km68c\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.442809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.442778 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:16.868121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.868089 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:37:16.869164 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:37:16.869130 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40d70f1_fe7b_4f62_8815_204e25ea44d8.slice/crio-7d81da4519dd0a578c0c276aac1591495a7f595922b90746500950ed5812555a WatchSource:0}: Error finding container 7d81da4519dd0a578c0c276aac1591495a7f595922b90746500950ed5812555a: Status 404 returned error can't find the container with id 7d81da4519dd0a578c0c276aac1591495a7f595922b90746500950ed5812555a Nov 02 21:37:16.871725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.871693 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:16.871822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.871758 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:16.871822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.871800 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:16.963096 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:16.963067 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" event={"ID":"c40d70f1-fe7b-4f62-8815-204e25ea44d8","Type":"ContainerStarted","Data":"7d81da4519dd0a578c0c276aac1591495a7f595922b90746500950ed5812555a"} Nov 02 21:37:17.044848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:17.044815 5859 patch_prober.go:29] interesting pod/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:17.045249 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:17.044872 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:17.972094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:17.972059 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" event={"ID":"c40d70f1-fe7b-4f62-8815-204e25ea44d8","Type":"ContainerStarted","Data":"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51"} Nov 02 21:37:18.055621 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.055583 5859 patch_prober.go:29] interesting pod/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:18.056162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.056047 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:18.182157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.182109 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podStartSLOduration=2.182097086 podStartE2EDuration="2.182097086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:18.180167591 +0000 UTC m=+1677.728417372" watchObservedRunningTime="2026-11-02 21:37:18.182097086 +0000 UTC m=+1677.730346866" Nov 02 21:37:18.454425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.454385 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:18.456807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.456777 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:37:18.456943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:18.456830 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 21:37:19.044380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:19.044346 5859 patch_prober.go:29] interesting pod/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:19.044664 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:19.044634 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:19.443318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:19.443279 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:37:19.443880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:19.443849 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 21:37:20.048567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:20.048536 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:37:20.048961 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:20.048930 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:37:20.049513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:20.049490 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:37:20.444031 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:20.444000 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:37:20.444499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:20.444060 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 21:37:21.443380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:21.443293 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:37:21.443380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:21.443348 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 21:37:22.443804 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:22.443767 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 21:37:22.444310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:22.443824 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 21:37:23.614145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:23.614109 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:23.614654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:23.614632 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:23.615679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:23.615662 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:37:25.955679 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:37:25.955638 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:37:31.085867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.085800 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:37:31.091278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.091259 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.102732 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.102708 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.102852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.102745 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.102852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.102837 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.102979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.102921 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.103038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.103009 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.103090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.103064 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.103090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.103081 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xts\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.103194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.103119 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.103253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.103234 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.204571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204474 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.204571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204534 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.204874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204628 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.204874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204690 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.204874 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204715 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xts\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.205061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.204929 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.205501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.205479 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.205765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.205746 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206137 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206191 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206260 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206301 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206326 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.206889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.206868 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.208008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.207926 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.208945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.208919 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.268530 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.268459 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.272156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.272125 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xts\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.295690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.295636 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:37:31.408584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.408502 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:31.837148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.837117 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:37:31.839371 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:37:31.839339 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2413f1e_8482_4bb8_8b9a_41ade9e44d35.slice/crio-fda1ca64a52c58c48f6284a09c365054d1ff31a494fbaee61788145db2a423e6 WatchSource:0}: Error finding container fda1ca64a52c58c48f6284a09c365054d1ff31a494fbaee61788145db2a423e6: Status 404 returned error can't find the container with id fda1ca64a52c58c48f6284a09c365054d1ff31a494fbaee61788145db2a423e6 Nov 02 21:37:31.842326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.842287 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:31.842415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.842383 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:31.842456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:31.842423 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:32.119137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.119046 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" event={"ID":"c2413f1e-8482-4bb8-8b9a-41ade9e44d35","Type":"ContainerStarted","Data":"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b"} Nov 02 21:37:32.119137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.119119 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" event={"ID":"c2413f1e-8482-4bb8-8b9a-41ade9e44d35","Type":"ContainerStarted","Data":"fda1ca64a52c58c48f6284a09c365054d1ff31a494fbaee61788145db2a423e6"} Nov 02 21:37:32.239600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.239544 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podStartSLOduration=1.239529057 podStartE2EDuration="1.239529057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:32.238476615 +0000 UTC m=+1691.786726396" watchObservedRunningTime="2026-11-02 21:37:32.239529057 +0000 UTC m=+1691.787778837" Nov 02 21:37:32.409750 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.409662 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:32.411154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.411117 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:32.411294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:32.411186 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:33.414982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:33.411282 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:33.414982 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:33.411342 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:34.412280 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:34.412243 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:34.412549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:34.412520 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:35.156022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.155992 5859 generic.go:356] "Generic (PLEG): container finished" podID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerID="21d1d32b18611d5ae53df600c9199c5639938bc07cabb58cbcd0f715442465a0" exitCode=0 Nov 02 21:37:35.156464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.156060 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerDied","Data":"21d1d32b18611d5ae53df600c9199c5639938bc07cabb58cbcd0f715442465a0"} Nov 02 21:37:35.156464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.156451 5859 scope.go:122] "RemoveContainer" containerID="21d1d32b18611d5ae53df600c9199c5639938bc07cabb58cbcd0f715442465a0" Nov 02 21:37:35.157941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.157915 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:35.158033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.158008 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:35.158073 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.158039 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:35.418038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.416390 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:35.418038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:35.416457 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:36.168779 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.168746 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerStarted","Data":"3f476694ad1e887d16eddb5d4f37c338fa23bf46e292a8a229f35f9063e8355a"} Nov 02 21:37:36.427032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.423315 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:36.427032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.423376 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:36.557903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.557860 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:37:36.559065 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.559031 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:37:36.559167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:36.559083 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 21:37:37.423974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:37.423290 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 21:37:37.423974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:37.423350 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 21:37:37.558311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:37.558269 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:37:37.558512 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:37.558375 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 21:37:37.559887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:37.559863 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:37:38.437851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:38.435563 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:38.437851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:38.436171 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:38.437851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:38.437360 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:37:38.557688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:38.557657 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:37:38.557873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:38.557709 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 21:37:39.557537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:39.557502 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:37:39.557932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:39.557567 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 21:37:40.557807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:40.557770 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 21:37:40.558320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:40.557841 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 21:37:41.562065 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:41.562029 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:37:41.563226 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:37:41.563207 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:37:55.945858 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:37:55.945817 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:38:06.370142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:06.370072 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:38:06.370640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:06.370273 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" containerID="cri-o://239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022" gracePeriod=30 Nov 02 21:38:09.612489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:09.612454 5859 patch_prober.go:29] interesting pod/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:09.612886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:09.612509 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:11.643275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.643251 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:38:11.730100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730049 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730109 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730147 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730228 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730301 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730423 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730515 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730462 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730515 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730478 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:11.730614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730530 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730568 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\") pod \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\" (UID: \"e325eb51-8b55-4d3b-99e6-2327b3031b7c\") " Nov 02 21:38:11.730614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730596 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:11.730761 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730614 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:11.730839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.730818 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:11.731156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731133 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:11.731399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731304 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.731399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731329 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.731399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731343 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.731399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731358 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.731399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.731372 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.732511 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.732491 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:11.733002 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.732984 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "kube-api-access-5hd8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:11.733070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.733049 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:11.733323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.733308 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo" pod "e325eb51-8b55-4d3b-99e6-2327b3031b7c" (UID: "e325eb51-8b55-4d3b-99e6-2327b3031b7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:11.832433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.832406 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hd8r\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-kube-api-access-5hd8r\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.832433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.832432 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.832662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.832442 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.832662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:11.832451 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325eb51-8b55-4d3b-99e6-2327b3031b7c-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:12.441185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.441151 5859 generic.go:356] "Generic (PLEG): container finished" podID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerID="239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022" exitCode=0 Nov 02 21:38:12.441360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.441208 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" Nov 02 21:38:12.441360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.441232 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerDied","Data":"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022"} Nov 02 21:38:12.441360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.441288 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq" event={"ID":"e325eb51-8b55-4d3b-99e6-2327b3031b7c","Type":"ContainerDied","Data":"3b9ebb3e23745fbebeb3e32b4f71d9225da8475949aac652e354de455ee4049e"} Nov 02 21:38:12.441360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.441311 5859 scope.go:122] "RemoveContainer" containerID="239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022" Nov 02 21:38:12.452341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.452324 5859 scope.go:122] "RemoveContainer" containerID="9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc" Nov 02 21:38:12.461388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.461367 5859 scope.go:122] "RemoveContainer" containerID="239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022" Nov 02 21:38:12.461642 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:12.461622 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022\": container with ID starting with 239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022 not found: ID does not exist" containerID="239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022" Nov 02 21:38:12.461686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.461650 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022"} err="failed to get container status \"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022\": rpc error: code = NotFound desc = could not find container \"239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022\": container with ID starting with 239bf1f736da3985b73ed75ed534f18841ffdfbfebc6ddd3f61b27f3b17d3022 not found: ID does not exist" Nov 02 21:38:12.461686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.461668 5859 scope.go:122] "RemoveContainer" containerID="9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc" Nov 02 21:38:12.461903 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:12.461882 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc\": container with ID starting with 9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc not found: ID does not exist" containerID="9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc" Nov 02 21:38:12.461989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.461903 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc"} err="failed to get container status \"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc\": rpc error: code = NotFound desc = could not find container \"9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc\": container with ID starting with 9d98634578a4d8c5d96ef21b32023db8a9afeca54de930c73a92de39b15ec2cc not found: ID does not exist" Nov 02 21:38:12.614604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.614582 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:38:12.664901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:12.664877 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-682116-openshift-default-7c69cbb989-tn4zq"] Nov 02 21:38:13.793262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:13.793230 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:38:13.899475 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:13.899444 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:38:14.570937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:14.570902 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" path="/var/lib/kubelet/pods/e325eb51-8b55-4d3b-99e6-2327b3031b7c/volumes" Nov 02 21:38:18.412880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:18.412845 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:38:18.521387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:18.521357 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:38:18.793902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:18.793811 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" podUID="19119e0c-3768-4860-83fd-35addee2a3e3" containerName="payload-processing" containerID="cri-o://631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8" gracePeriod=25 Nov 02 21:38:18.900047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:18.900009 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" podUID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" containerName="payload-pre-processing" containerID="cri-o://d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d" gracePeriod=25 Nov 02 21:38:19.086268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.086024 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:38:19.180678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.180657 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:38:19.184253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184233 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\" (UniqueName: \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\") pod \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " Nov 02 21:38:19.184359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184347 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\") pod \"19119e0c-3768-4860-83fd-35addee2a3e3\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " Nov 02 21:38:19.184409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184383 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\" (UniqueName: \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\") pod \"19119e0c-3768-4860-83fd-35addee2a3e3\" (UID: \"19119e0c-3768-4860-83fd-35addee2a3e3\") " Nov 02 21:38:19.184456 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184411 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\") pod \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\" (UID: \"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4\") " Nov 02 21:38:19.184693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184670 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume" pod "19119e0c-3768-4860-83fd-35addee2a3e3" (UID: "19119e0c-3768-4860-83fd-35addee2a3e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:19.184838 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.184816 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume" pod "e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" (UID: "e2185fe8-4ec1-4ca6-ad91-907ab685c4e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:19.185056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.185040 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19119e0c-3768-4860-83fd-35addee2a3e3-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:19.185104 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.185058 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:19.186185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.186167 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn" pod "e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" (UID: "e2185fe8-4ec1-4ca6-ad91-907ab685c4e4"). InnerVolumeSpecName "kube-api-access-smmdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:19.186380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.186360 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds" pod "19119e0c-3768-4860-83fd-35addee2a3e3" (UID: "19119e0c-3768-4860-83fd-35addee2a3e3"). InnerVolumeSpecName "kube-api-access-wlxds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:19.286535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.286508 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlxds\" (UniqueName: \"kubernetes.io/projected/19119e0c-3768-4860-83fd-35addee2a3e3-kube-api-access-wlxds\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:19.286535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.286534 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmdn\" (UniqueName: \"kubernetes.io/projected/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4-kube-api-access-smmdn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:19.582790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.582763 5859 generic.go:356] "Generic (PLEG): container finished" podID="19119e0c-3768-4860-83fd-35addee2a3e3" containerID="631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8" exitCode=0 Nov 02 21:38:19.583259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.582816 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" Nov 02 21:38:19.583259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.582853 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" event={"ID":"19119e0c-3768-4860-83fd-35addee2a3e3","Type":"ContainerDied","Data":"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8"} Nov 02 21:38:19.583259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.582900 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc" event={"ID":"19119e0c-3768-4860-83fd-35addee2a3e3","Type":"ContainerDied","Data":"1a513125c8f1c949020b45b22a1841b759f54cce43db21d219dfda5575626dd2"} Nov 02 21:38:19.583259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.582930 5859 scope.go:122] "RemoveContainer" containerID="631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8" Nov 02 21:38:19.584615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.584581 5859 generic.go:356] "Generic (PLEG): container finished" podID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" containerID="d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d" exitCode=0 Nov 02 21:38:19.584747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.584644 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" Nov 02 21:38:19.584747 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.584646 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" event={"ID":"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4","Type":"ContainerDied","Data":"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d"} Nov 02 21:38:19.585019 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.584768 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg" event={"ID":"e2185fe8-4ec1-4ca6-ad91-907ab685c4e4","Type":"ContainerDied","Data":"30e776aa63b9862984420d1415d13900714df9decad88660383b8583963aba74"} Nov 02 21:38:19.598101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.597834 5859 scope.go:122] "RemoveContainer" containerID="631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8" Nov 02 21:38:19.598335 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:19.598306 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8\": container with ID starting with 631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8 not found: ID does not exist" containerID="631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8" Nov 02 21:38:19.598458 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.598342 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8"} err="failed to get container status \"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8\": rpc error: code = NotFound desc = could not find container \"631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8\": container with ID starting with 631417313df68f326c0fcc1a4013188bed65e94759a0d3330b440fd31076bba8 not found: ID does not exist" Nov 02 21:38:19.598458 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.598366 5859 scope.go:122] "RemoveContainer" containerID="d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d" Nov 02 21:38:19.608282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.608260 5859 scope.go:122] "RemoveContainer" containerID="d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d" Nov 02 21:38:19.608641 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:19.608611 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d\": container with ID starting with d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d not found: ID does not exist" containerID="d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d" Nov 02 21:38:19.608757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.608647 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d"} err="failed to get container status \"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d\": rpc error: code = NotFound desc = could not find container \"d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d\": container with ID starting with d0f0c16615cc8a912d9cf18149d844cfba905374f525eff4955d35f1ffc75e2d not found: ID does not exist" Nov 02 21:38:19.748498 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.748476 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:38:19.754052 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.754029 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6606aa-794fdbbcbf-rntdc"] Nov 02 21:38:19.866494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.866402 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:38:19.881109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:19.881067 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6606aa-5d764c756c-d5mwg"] Nov 02 21:38:20.573145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:20.573114 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19119e0c-3768-4860-83fd-35addee2a3e3" path="/var/lib/kubelet/pods/19119e0c-3768-4860-83fd-35addee2a3e3/volumes" Nov 02 21:38:20.573551 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:20.573538 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" path="/var/lib/kubelet/pods/e2185fe8-4ec1-4ca6-ad91-907ab685c4e4/volumes" Nov 02 21:38:22.730384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:22.730348 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:38:22.730835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:22.730560 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" podUID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" containerName="maas-api" containerID="cri-o://081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba" gracePeriod=30 Nov 02 21:38:23.022921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.022899 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:38:23.144151 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.144123 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") pod \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " Nov 02 21:38:23.144613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.144164 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\") pod \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " Nov 02 21:38:23.144613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.144243 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\" (UniqueName: \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\") pod \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " Nov 02 21:38:23.144613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.144299 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\") pod \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\" (UID: \"c91e93eb-2f0e-4021-ad85-5e7c0018be87\") " Nov 02 21:38:23.145193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.145171 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca" pod "c91e93eb-2f0e-4021-ad85-5e7c0018be87" (UID: "c91e93eb-2f0e-4021-ad85-5e7c0018be87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:23.147023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.146998 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls" pod "c91e93eb-2f0e-4021-ad85-5e7c0018be87" (UID: "c91e93eb-2f0e-4021-ad85-5e7c0018be87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:38:23.147513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.147476 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96" pod "c91e93eb-2f0e-4021-ad85-5e7c0018be87" (UID: "c91e93eb-2f0e-4021-ad85-5e7c0018be87"). InnerVolumeSpecName "kube-api-access-s8q96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:23.148011 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.147943 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs" pod "c91e93eb-2f0e-4021-ad85-5e7c0018be87" (UID: "c91e93eb-2f0e-4021-ad85-5e7c0018be87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:38:23.246141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.246071 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.246141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.246103 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c91e93eb-2f0e-4021-ad85-5e7c0018be87-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.246141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.246117 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8q96\" (UniqueName: \"kubernetes.io/projected/c91e93eb-2f0e-4021-ad85-5e7c0018be87-kube-api-access-s8q96\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.246141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.246131 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c91e93eb-2f0e-4021-ad85-5e7c0018be87-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.413596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.413558 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" podUID="2f486811-c89f-4781-b561-3bc8972e759f" containerName="payload-processing" containerID="cri-o://80e430bef30547fea3df06a003fe4c38ec00f92aff75c3584f88b63857ebd2e9" gracePeriod=25 Nov 02 21:38:23.522148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.522087 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" podUID="269174e2-caa4-4c25-a58e-30dbd9eed84f" containerName="payload-pre-processing" containerID="cri-o://ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848" gracePeriod=25 Nov 02 21:38:23.618428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.618387 5859 generic.go:356] "Generic (PLEG): container finished" podID="2f486811-c89f-4781-b561-3bc8972e759f" containerID="80e430bef30547fea3df06a003fe4c38ec00f92aff75c3584f88b63857ebd2e9" exitCode=0 Nov 02 21:38:23.618585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.618486 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" event={"ID":"2f486811-c89f-4781-b561-3bc8972e759f","Type":"ContainerDied","Data":"80e430bef30547fea3df06a003fe4c38ec00f92aff75c3584f88b63857ebd2e9"} Nov 02 21:38:23.620335 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.620304 5859 generic.go:356] "Generic (PLEG): container finished" podID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" containerID="081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba" exitCode=0 Nov 02 21:38:23.620484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.620350 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" Nov 02 21:38:23.620484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.620389 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" event={"ID":"c91e93eb-2f0e-4021-ad85-5e7c0018be87","Type":"ContainerDied","Data":"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba"} Nov 02 21:38:23.620484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.620432 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4" event={"ID":"c91e93eb-2f0e-4021-ad85-5e7c0018be87","Type":"ContainerDied","Data":"6bb466e23de7ca2303922b61e348d673ae8a1d7d373da242495488fa23a3cf14"} Nov 02 21:38:23.620484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.620460 5859 scope.go:122] "RemoveContainer" containerID="081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba" Nov 02 21:38:23.636888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.636865 5859 scope.go:122] "RemoveContainer" containerID="081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba" Nov 02 21:38:23.637280 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:23.637252 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba\": container with ID starting with 081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba not found: ID does not exist" containerID="081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba" Nov 02 21:38:23.637386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.637288 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba"} err="failed to get container status \"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba\": rpc error: code = NotFound desc = could not find container \"081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba\": container with ID starting with 081fbc2008852f35e5c14028b2b515e94c9eb42c1a988c7d0182adba9dc248ba not found: ID does not exist" Nov 02 21:38:23.653152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.653133 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:38:23.752396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.752364 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\") pod \"2f486811-c89f-4781-b561-3bc8972e759f\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " Nov 02 21:38:23.753084 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.753055 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\" (UniqueName: \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\") pod \"2f486811-c89f-4781-b561-3bc8972e759f\" (UID: \"2f486811-c89f-4781-b561-3bc8972e759f\") " Nov 02 21:38:23.753505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.753478 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume" pod "2f486811-c89f-4781-b561-3bc8972e759f" (UID: "2f486811-c89f-4781-b561-3bc8972e759f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:23.754203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.754182 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f486811-c89f-4781-b561-3bc8972e759f-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.755718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.755688 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4" pod "2f486811-c89f-4781-b561-3bc8972e759f" (UID: "2f486811-c89f-4781-b561-3bc8972e759f"). InnerVolumeSpecName "kube-api-access-jzzm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:23.892976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.884015 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzzm4\" (UniqueName: \"kubernetes.io/projected/2f486811-c89f-4781-b561-3bc8972e759f-kube-api-access-jzzm4\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.914382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.914357 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:38:23.935491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.935461 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:38:23.938307 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.938280 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6606aa-5ff64dc49b-rz4c4"] Nov 02 21:38:23.984784 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.984762 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\") pod \"269174e2-caa4-4c25-a58e-30dbd9eed84f\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " Nov 02 21:38:23.985023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.985005 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\" (UniqueName: \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\") pod \"269174e2-caa4-4c25-a58e-30dbd9eed84f\" (UID: \"269174e2-caa4-4c25-a58e-30dbd9eed84f\") " Nov 02 21:38:23.985196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.985178 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume" pod "269174e2-caa4-4c25-a58e-30dbd9eed84f" (UID: "269174e2-caa4-4c25-a58e-30dbd9eed84f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:23.985503 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.985490 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/269174e2-caa4-4c25-a58e-30dbd9eed84f-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:23.986929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:23.986905 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw" pod "269174e2-caa4-4c25-a58e-30dbd9eed84f" (UID: "269174e2-caa4-4c25-a58e-30dbd9eed84f"). InnerVolumeSpecName "kube-api-access-prcjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:24.087117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.087083 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prcjw\" (UniqueName: \"kubernetes.io/projected/269174e2-caa4-4c25-a58e-30dbd9eed84f-kube-api-access-prcjw\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:24.571325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.571295 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" path="/var/lib/kubelet/pods/c91e93eb-2f0e-4021-ad85-5e7c0018be87/volumes" Nov 02 21:38:24.630570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.630535 5859 generic.go:356] "Generic (PLEG): container finished" podID="269174e2-caa4-4c25-a58e-30dbd9eed84f" containerID="ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848" exitCode=0 Nov 02 21:38:24.630702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.630631 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" Nov 02 21:38:24.630702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.630655 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" event={"ID":"269174e2-caa4-4c25-a58e-30dbd9eed84f","Type":"ContainerDied","Data":"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848"} Nov 02 21:38:24.630814 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.630714 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj" event={"ID":"269174e2-caa4-4c25-a58e-30dbd9eed84f","Type":"ContainerDied","Data":"6528bca5c5fc2be50f6b5be35d03c82ffb04a46f147dc5f7becea31d430439ea"} Nov 02 21:38:24.630814 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.630745 5859 scope.go:122] "RemoveContainer" containerID="ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848" Nov 02 21:38:24.632481 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.632456 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" event={"ID":"2f486811-c89f-4781-b561-3bc8972e759f","Type":"ContainerDied","Data":"3d5bcc4c4cbdced49cd223bd8abcabc734a29ee3725914d49507c8b06d26be7f"} Nov 02 21:38:24.632730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.632490 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq" Nov 02 21:38:24.640995 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.640928 5859 scope.go:122] "RemoveContainer" containerID="ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848" Nov 02 21:38:24.641279 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:24.641260 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848\": container with ID starting with ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848 not found: ID does not exist" containerID="ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848" Nov 02 21:38:24.641331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.641287 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848"} err="failed to get container status \"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848\": rpc error: code = NotFound desc = could not find container \"ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848\": container with ID starting with ff5a5d51b0fd7d9dc4ab94115d8cf7a0dd721f77749a0b1144f427509c006848 not found: ID does not exist" Nov 02 21:38:24.641331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.641305 5859 scope.go:122] "RemoveContainer" containerID="80e430bef30547fea3df06a003fe4c38ec00f92aff75c3584f88b63857ebd2e9" Nov 02 21:38:24.800201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.800171 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:38:24.808966 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.808935 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70efbbab-79796576df-bl5pq"] Nov 02 21:38:24.920891 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.920795 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:38:24.925234 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:24.925208 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70efbbab-9788db44c-c86mj"] Nov 02 21:38:25.945290 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:25.945251 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:38:26.570702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:26.570671 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269174e2-caa4-4c25-a58e-30dbd9eed84f" path="/var/lib/kubelet/pods/269174e2-caa4-4c25-a58e-30dbd9eed84f/volumes" Nov 02 21:38:26.571077 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:26.571063 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f486811-c89f-4781-b561-3bc8972e759f" path="/var/lib/kubelet/pods/2f486811-c89f-4781-b561-3bc8972e759f/volumes" Nov 02 21:38:27.463977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:27.463931 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:38:27.464395 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:27.464163 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" containerID="cri-o://0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5" gracePeriod=30 Nov 02 21:38:32.722598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.722576 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:38:32.754625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.754566 5859 generic.go:356] "Generic (PLEG): container finished" podID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerID="0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5" exitCode=0 Nov 02 21:38:32.754625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.754602 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerDied","Data":"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5"} Nov 02 21:38:32.754802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.754651 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" event={"ID":"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635","Type":"ContainerDied","Data":"f6e26de9da1746f6e0d5382a795ff3f0b0e53e1ac9ab9239eaca6a2a568d5080"} Nov 02 21:38:32.754802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.754681 5859 scope.go:122] "RemoveContainer" containerID="0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5" Nov 02 21:38:32.754802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.754606 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk" Nov 02 21:38:32.766207 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.766189 5859 scope.go:122] "RemoveContainer" containerID="9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453" Nov 02 21:38:32.774699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.774679 5859 scope.go:122] "RemoveContainer" containerID="0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5" Nov 02 21:38:32.774972 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:32.774934 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5\": container with ID starting with 0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5 not found: ID does not exist" containerID="0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5" Nov 02 21:38:32.775081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.774978 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5"} err="failed to get container status \"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5\": rpc error: code = NotFound desc = could not find container \"0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5\": container with ID starting with 0b04c0f672561da64cf566c1f76d08ab13668db01e38c5211d998eaa6a4bd9b5 not found: ID does not exist" Nov 02 21:38:32.775081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.774999 5859 scope.go:122] "RemoveContainer" containerID="9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453" Nov 02 21:38:32.775272 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:32.775253 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453\": container with ID starting with 9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453 not found: ID does not exist" containerID="9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453" Nov 02 21:38:32.775324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.775282 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453"} err="failed to get container status \"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453\": rpc error: code = NotFound desc = could not find container \"9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453\": container with ID starting with 9c1d9c5d5677b890d8d348e526de0c0aa8099f72cfdb68e7e2880123eb589453 not found: ID does not exist" Nov 02 21:38:32.911688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.911662 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.911852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.911763 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.911852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.911813 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.911983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.911926 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.911983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.911946 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912000 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912039 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912085 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912098 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.912248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912146 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912147 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.912248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912228 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\") pod \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\" (UID: \"dd3da1d4-bc4c-48fb-b1ea-49bb783c0635\") " Nov 02 21:38:32.912464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912437 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912527 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912788 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912808 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912824 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912839 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.912877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.912854 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.914197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.914147 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:32.914308 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.914285 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:32.914516 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.914495 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.917136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:32.917111 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz" pod "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" (UID: "dd3da1d4-bc4c-48fb-b1ea-49bb783c0635"). InnerVolumeSpecName "kube-api-access-qd4zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:33.014591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.014526 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:33.014591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.014552 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:33.014591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.014561 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:33.014591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.014569 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd4zz\" (UniqueName: \"kubernetes.io/projected/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635-kube-api-access-qd4zz\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:33.181044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.181008 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:38:33.183495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:33.183471 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6606aa-openshift-default-57f996cd6c-mz4lk"] Nov 02 21:38:34.570468 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:38:34.570435 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" path="/var/lib/kubelet/pods/dd3da1d4-bc4c-48fb-b1ea-49bb783c0635/volumes" Nov 02 21:38:55.945050 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:38:55.945008 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:39:24.991161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991123 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:39:24.991726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991711 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991727 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991737 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991742 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991754 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2f486811-c89f-4781-b561-3bc8972e759f" containerName="payload-processing" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991759 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f486811-c89f-4781-b561-3bc8972e759f" containerName="payload-processing" Nov 02 21:39:24.991768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991768 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" containerName="payload-pre-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991774 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" containerName="payload-pre-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991798 5859 cpu_manager.go:414] "containerMap: removing container" podUID="269174e2-caa4-4c25-a58e-30dbd9eed84f" containerName="payload-pre-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991804 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269174e2-caa4-4c25-a58e-30dbd9eed84f" containerName="payload-pre-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991825 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991830 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991838 5859 cpu_manager.go:414] "containerMap: removing container" podUID="19119e0c-3768-4860-83fd-35addee2a3e3" containerName="payload-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991842 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19119e0c-3768-4860-83fd-35addee2a3e3" containerName="payload-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991851 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" containerName="maas-api" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991857 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" containerName="maas-api" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991943 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991964 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="19119e0c-3768-4860-83fd-35addee2a3e3" containerName="payload-processing" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991974 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91e93eb-2f0e-4021-ad85-5e7c0018be87" containerName="maas-api" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991981 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:39:24.991984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991987 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3da1d4-bc4c-48fb-b1ea-49bb783c0635" containerName="istio-proxy" Nov 02 21:39:24.992384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991993 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f486811-c89f-4781-b561-3bc8972e759f" containerName="payload-processing" Nov 02 21:39:24.992384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.991999 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2185fe8-4ec1-4ca6-ad91-907ab685c4e4" containerName="payload-pre-processing" Nov 02 21:39:24.992384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.992007 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="269174e2-caa4-4c25-a58e-30dbd9eed84f" containerName="payload-pre-processing" Nov 02 21:39:24.996673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:24.996652 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.049864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.049837 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:39:25.097159 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.097136 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e12f99-serving-cert\"" Nov 02 21:39:25.133843 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.133802 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9pg\" (UniqueName: \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.134090 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.134064 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.134182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.134114 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.134182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.134162 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.247578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.247487 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.248043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.248015 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9pg\" (UniqueName: \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.248245 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:39:25.248219 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-serving-cert: secret "maas-api-e2e-ipp-a-e12f99-serving-cert" not found Nov 02 21:39:25.248433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.248409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.248536 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:39:25.248521 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls podName:7fb46b67-2a9c-4903-a3ff-5384c9fb8187 nodeName:}" failed. No retries permitted until 2026-11-02 21:39:25.748425819 +0000 UTC m=+1805.296675604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls") pod "maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" (UID: "7fb46b67-2a9c-4903-a3ff-5384c9fb8187") : secret "maas-api-e2e-ipp-a-e12f99-serving-cert" not found Nov 02 21:39:25.248683 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.248660 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.249301 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.249275 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.253479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.253456 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.308438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.308405 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9pg\" (UniqueName: \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.755722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.755684 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.758103 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.758079 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.910167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:25.910135 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:25.945883 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:39:25.945843 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:39:26.289292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:26.289265 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:39:26.290495 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:39:26.290471 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb46b67_2a9c_4903_a3ff_5384c9fb8187.slice/crio-463a42d4e03b92d59091b97c1500b67a0e45f452ba8e87e1917c95a3f1f41aad WatchSource:0}: Error finding container 463a42d4e03b92d59091b97c1500b67a0e45f452ba8e87e1917c95a3f1f41aad: Status 404 returned error can't find the container with id 463a42d4e03b92d59091b97c1500b67a0e45f452ba8e87e1917c95a3f1f41aad Nov 02 21:39:27.127000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:27.126964 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" event={"ID":"7fb46b67-2a9c-4903-a3ff-5384c9fb8187","Type":"ContainerStarted","Data":"463a42d4e03b92d59091b97c1500b67a0e45f452ba8e87e1917c95a3f1f41aad"} Nov 02 21:39:28.134940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:28.134903 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" event={"ID":"7fb46b67-2a9c-4903-a3ff-5384c9fb8187","Type":"ContainerStarted","Data":"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd"} Nov 02 21:39:28.135415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:28.135130 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:28.251073 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:28.251025 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" podStartSLOduration=3.256237719 podStartE2EDuration="4.251011844s" totalImagesPullingTime="994.774125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:28.250546106 +0000 UTC m=+1807.798795888" watchObservedRunningTime="2026-11-02 21:39:28.251011844 +0000 UTC m=+1807.799261623" Nov 02 21:39:34.157304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:39:34.157194 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:39:55.944935 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:39:55.944895 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:40:05.978069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:05.978034 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:40:05.978468 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:05.978213 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" podUID="cfe05938-b807-48e3-91bd-3b19f4850aad" containerName="maas-api" containerID="cri-o://f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be" gracePeriod=30 Nov 02 21:40:06.281180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.281155 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:40:06.344151 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344119 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\") pod \"cfe05938-b807-48e3-91bd-3b19f4850aad\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " Nov 02 21:40:06.344268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344243 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\" (UniqueName: \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\") pod \"cfe05938-b807-48e3-91bd-3b19f4850aad\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " Nov 02 21:40:06.344318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344303 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\") pod \"cfe05938-b807-48e3-91bd-3b19f4850aad\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " Nov 02 21:40:06.344446 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344424 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") pod \"cfe05938-b807-48e3-91bd-3b19f4850aad\" (UID: \"cfe05938-b807-48e3-91bd-3b19f4850aad\") " Nov 02 21:40:06.344634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344602 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca" pod "cfe05938-b807-48e3-91bd-3b19f4850aad" (UID: "cfe05938-b807-48e3-91bd-3b19f4850aad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:06.344972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.344940 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe05938-b807-48e3-91bd-3b19f4850aad-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:06.346482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.346461 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc" pod "cfe05938-b807-48e3-91bd-3b19f4850aad" (UID: "cfe05938-b807-48e3-91bd-3b19f4850aad"). InnerVolumeSpecName "kube-api-access-bnbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:06.346569 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.346482 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs" pod "cfe05938-b807-48e3-91bd-3b19f4850aad" (UID: "cfe05938-b807-48e3-91bd-3b19f4850aad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:06.346569 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.346498 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls" pod "cfe05938-b807-48e3-91bd-3b19f4850aad" (UID: "cfe05938-b807-48e3-91bd-3b19f4850aad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:06.432303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.432262 5859 generic.go:356] "Generic (PLEG): container finished" podID="cfe05938-b807-48e3-91bd-3b19f4850aad" containerID="f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be" exitCode=0 Nov 02 21:40:06.432482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.432379 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" event={"ID":"cfe05938-b807-48e3-91bd-3b19f4850aad","Type":"ContainerDied","Data":"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be"} Nov 02 21:40:06.432482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.432466 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" event={"ID":"cfe05938-b807-48e3-91bd-3b19f4850aad","Type":"ContainerDied","Data":"baa72711cdbb66fc94439dc7bf0039e134dac5c2e8421edb2e08639cb57e9c0e"} Nov 02 21:40:06.432563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.432496 5859 scope.go:122] "RemoveContainer" containerID="f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be" Nov 02 21:40:06.432665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.432651 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.449524 5859 scope.go:122] "RemoveContainer" containerID="f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.451261 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnbwc\" (UniqueName: \"kubernetes.io/projected/cfe05938-b807-48e3-91bd-3b19f4850aad-kube-api-access-bnbwc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.451296 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.451314 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe05938-b807-48e3-91bd-3b19f4850aad-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:40:06.451664 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be\": container with ID starting with f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be not found: ID does not exist" containerID="f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be" Nov 02 21:40:06.458248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.451698 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be"} err="failed to get container status \"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be\": rpc error: code = NotFound desc = could not find container \"f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be\": container with ID starting with f760adb7b9a28f24de36b328cc4da7a5e20a3f41e789c9794f4ef0ff5e6d31be not found: ID does not exist" Nov 02 21:40:06.599392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.599356 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:40:06.604443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:06.604419 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70efbbab-75695b9cf-hgkvd"] Nov 02 21:40:08.369120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.369090 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:40:08.369585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.369330 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" containerID="cri-o://d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51" gracePeriod=30 Nov 02 21:40:08.388938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.388908 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:40:08.389109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.389090 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" containerID="cri-o://a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b" gracePeriod=30 Nov 02 21:40:08.438571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.438539 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:08.438689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.438591 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:08.571372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.571340 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe05938-b807-48e3-91bd-3b19f4850aad" path="/var/lib/kubelet/pods/cfe05938-b807-48e3-91bd-3b19f4850aad/volumes" Nov 02 21:40:08.616899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.616855 5859 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:08.617088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:08.616927 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:13.664803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.664769 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:40:13.719268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.719245 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:40:13.734832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.734809 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.734932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.734839 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.734932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.734869 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.734932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.734899 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.734935 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735005 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735089 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.735257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735098 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735233 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735250 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.735423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735273 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735319 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735352 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735438 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735494 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735557 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735654 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735706 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\") pod \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\" (UID: \"c40d70f1-fe7b-4f62-8815-204e25ea44d8\") " Nov 02 21:40:13.735733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735441 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.735733 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735706 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.735942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735734 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.735942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.735768 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\") pod \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\" (UID: \"c2413f1e-8482-4bb8-8b9a-41ade9e44d35\") " Nov 02 21:40:13.736736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.736043 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.736736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.736159 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.736736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.736343 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:13.736736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.736518 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:13.737258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737232 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737264 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737282 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737297 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737311 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737324 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737338 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.737403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.737353 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.738145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738119 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.738258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738149 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.738443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738413 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "kube-api-access-t98nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.738443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738435 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.738567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738484 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.738567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738518 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.739022 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.738996 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "kube-api-access-z8xts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.739107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.739086 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo" pod "c40d70f1-fe7b-4f62-8815-204e25ea44d8" (UID: "c40d70f1-fe7b-4f62-8815-204e25ea44d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:13.739810 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.739790 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:13.739875 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.739821 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy" pod "c2413f1e-8482-4bb8-8b9a-41ade9e44d35" (UID: "c2413f1e-8482-4bb8-8b9a-41ade9e44d35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.838826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838797 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.838826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838827 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838840 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838854 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98nb\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-kube-api-access-t98nb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838866 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838878 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838889 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838902 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838914 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40d70f1-fe7b-4f62-8815-204e25ea44d8-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.839029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:13.838926 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xts\" (UniqueName: \"kubernetes.io/projected/c2413f1e-8482-4bb8-8b9a-41ade9e44d35-kube-api-access-z8xts\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:14.485044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.485018 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" Nov 02 21:40:14.485172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.485064 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" event={"ID":"c2413f1e-8482-4bb8-8b9a-41ade9e44d35","Type":"ContainerDied","Data":"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b"} Nov 02 21:40:14.485172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.484943 5859 generic.go:356] "Generic (PLEG): container finished" podID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerID="a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b" exitCode=0 Nov 02 21:40:14.485172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.485122 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx" event={"ID":"c2413f1e-8482-4bb8-8b9a-41ade9e44d35","Type":"ContainerDied","Data":"fda1ca64a52c58c48f6284a09c365054d1ff31a494fbaee61788145db2a423e6"} Nov 02 21:40:14.485172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.485146 5859 scope.go:122] "RemoveContainer" containerID="a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b" Nov 02 21:40:14.486860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.486835 5859 generic.go:356] "Generic (PLEG): container finished" podID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerID="d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51" exitCode=0 Nov 02 21:40:14.486933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.486882 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" Nov 02 21:40:14.486933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.486887 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" event={"ID":"c40d70f1-fe7b-4f62-8815-204e25ea44d8","Type":"ContainerDied","Data":"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51"} Nov 02 21:40:14.486933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.486927 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c" event={"ID":"c40d70f1-fe7b-4f62-8815-204e25ea44d8","Type":"ContainerDied","Data":"7d81da4519dd0a578c0c276aac1591495a7f595922b90746500950ed5812555a"} Nov 02 21:40:14.499164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.497193 5859 scope.go:122] "RemoveContainer" containerID="a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b" Nov 02 21:40:14.499164 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:40:14.497481 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b\": container with ID starting with a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b not found: ID does not exist" containerID="a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b" Nov 02 21:40:14.499164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.497522 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b"} err="failed to get container status \"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b\": rpc error: code = NotFound desc = could not find container \"a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b\": container with ID starting with a5d13858cf71addb0ef60740a1126a892af10e5a9469886ce5273c69ae2fb75b not found: ID does not exist" Nov 02 21:40:14.499164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.497546 5859 scope.go:122] "RemoveContainer" containerID="d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51" Nov 02 21:40:14.506794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.506774 5859 scope.go:122] "RemoveContainer" containerID="d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51" Nov 02 21:40:14.507079 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:40:14.507060 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51\": container with ID starting with d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51 not found: ID does not exist" containerID="d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51" Nov 02 21:40:14.507154 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.507086 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51"} err="failed to get container status \"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51\": rpc error: code = NotFound desc = could not find container \"d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51\": container with ID starting with d3893d2d54677f8e396c270d4ada9c036870031e70d9c50292908d121aa47f51 not found: ID does not exist" Nov 02 21:40:14.684057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.683987 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:40:14.736113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.736080 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-km68c"] Nov 02 21:40:14.848386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.848357 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:40:14.851693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:14.851670 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-5r9dx"] Nov 02 21:40:16.571315 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:16.571276 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" path="/var/lib/kubelet/pods/c2413f1e-8482-4bb8-8b9a-41ade9e44d35/volumes" Nov 02 21:40:16.571737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:16.571723 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" path="/var/lib/kubelet/pods/c40d70f1-fe7b-4f62-8815-204e25ea44d8/volumes" Nov 02 21:40:25.943577 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:40:25.943540 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:40:27.106734 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.106697 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:40:27.108279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.108250 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" Nov 02 21:40:27.108457 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.108442 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" Nov 02 21:40:27.108580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.108567 5859 cpu_manager.go:414] "containerMap: removing container" podUID="cfe05938-b807-48e3-91bd-3b19f4850aad" containerName="maas-api" Nov 02 21:40:27.108684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.108671 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe05938-b807-48e3-91bd-3b19f4850aad" containerName="maas-api" Nov 02 21:40:27.108813 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.108802 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:40:27.110726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.110704 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:40:27.110900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.110880 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" Nov 02 21:40:27.110900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.110897 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" Nov 02 21:40:27.111131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.111057 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40d70f1-fe7b-4f62-8815-204e25ea44d8" containerName="istio-proxy" Nov 02 21:40:27.111131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.111073 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe05938-b807-48e3-91bd-3b19f4850aad" containerName="maas-api" Nov 02 21:40:27.111131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.111087 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325eb51-8b55-4d3b-99e6-2327b3031b7c" containerName="istio-proxy" Nov 02 21:40:27.111131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.111098 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2413f1e-8482-4bb8-8b9a-41ade9e44d35" containerName="istio-proxy" Nov 02 21:40:27.124717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.121234 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.130682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.130630 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131036 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131225 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5xp\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131402 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131376 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131449 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131482 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131514 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131619 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.131721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.131704 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.205295 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.205260 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:40:27.233494 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.233454 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.233658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.233530 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5xp\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.234372 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.234344 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.233568 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250589 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250623 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250681 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250754 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.250835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.250826 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.251333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.251089 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.251333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.251174 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.251333 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.251266 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.251573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.251370 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.252037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.252014 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ce2453-openshift-default-dockercfg-fpvzk\"" Nov 02 21:40:27.253557 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.253526 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.253725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.253704 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.292056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.292022 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.292209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.292096 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5xp\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\") pod \"e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.449407 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.449313 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:27.877164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.877137 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:40:27.877847 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:40:27.877822 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc97b16c_7401_43eb_96a3_db052723e6cb.slice/crio-9363e2411ba89093f43e2ca3e3a1191a7c29a1fe61e13c16af113b92a35e991e WatchSource:0}: Error finding container 9363e2411ba89093f43e2ca3e3a1191a7c29a1fe61e13c16af113b92a35e991e: Status 404 returned error can't find the container with id 9363e2411ba89093f43e2ca3e3a1191a7c29a1fe61e13c16af113b92a35e991e Nov 02 21:40:27.880156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.880128 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:27.880216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.880186 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:27.880255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:27.880213 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:28.607146 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:28.607105 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerStarted","Data":"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd"} Nov 02 21:40:28.607548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:28.607166 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerStarted","Data":"9363e2411ba89093f43e2ca3e3a1191a7c29a1fe61e13c16af113b92a35e991e"} Nov 02 21:40:28.828313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:28.828264 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podStartSLOduration=1.8282499859999999 podStartE2EDuration="1.828249986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:28.776353716 +0000 UTC m=+1868.324603520" watchObservedRunningTime="2026-11-02 21:40:28.828249986 +0000 UTC m=+1868.376499767" Nov 02 21:40:29.202669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.202640 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:40:29.207107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.207087 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.268025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.266703 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:40:29.309974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.309866 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.310177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310057 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2qx\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.310177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310109 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.310177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310124 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.310317 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310262 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.310317 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310285 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.315450 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310482 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.315450 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310627 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.315450 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.310674 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.349102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.349052 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-387f88-openshift-default-dockercfg-7hzhv\"" Nov 02 21:40:29.412005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.411933 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412088 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412162 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2qx\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412218 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412241 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412319 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412347 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412406 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412493 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412622 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412579 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412819 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412882 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.412997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.412936 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.413260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.413237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.414907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.414865 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.415122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.415105 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.450248 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.450225 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:29.455535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.455479 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:29.470629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.470609 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.470823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.470803 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2qx\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\") pod \"e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.520876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.520846 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:29.627514 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.627480 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:29.628696 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.628674 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:40:29.952843 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.952814 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:40:29.953495 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:40:29.953469 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bebdf28_778c_4b94_babf_8162bf6ebe8c.slice/crio-951979ea28f0571dfe94ce48bda796ee877f8ef01b6a73c7f24dce450966c201 WatchSource:0}: Error finding container 951979ea28f0571dfe94ce48bda796ee877f8ef01b6a73c7f24dce450966c201: Status 404 returned error can't find the container with id 951979ea28f0571dfe94ce48bda796ee877f8ef01b6a73c7f24dce450966c201 Nov 02 21:40:29.956180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.956147 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:29.956412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.956213 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:29.956412 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:29.956240 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:30.634005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:30.633968 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerStarted","Data":"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24"} Nov 02 21:40:30.634397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:30.634025 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerStarted","Data":"951979ea28f0571dfe94ce48bda796ee877f8ef01b6a73c7f24dce450966c201"} Nov 02 21:40:30.751891 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:30.751847 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podStartSLOduration=1.75183483 podStartE2EDuration="1.75183483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:30.750004227 +0000 UTC m=+1870.298254046" watchObservedRunningTime="2026-11-02 21:40:30.75183483 +0000 UTC m=+1870.300084610" Nov 02 21:40:31.521192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:31.521154 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:31.526917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:31.526894 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:31.646118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:31.646086 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:31.649918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:31.649892 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:40:39.058543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.058506 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:40:39.063018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.062921 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.116072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.116044 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:40:39.146563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.146537 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.146717 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.146682 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjv4z\" (UniqueName: \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.163470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.163444 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ce2453-dockercfg-fs59l\"" Nov 02 21:40:39.163470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.163448 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ce2453\"" Nov 02 21:40:39.248337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.248307 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjv4z\" (UniqueName: \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.248501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.248485 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.249172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.249154 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.305205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.305170 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjv4z\" (UniqueName: \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\") pod \"payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.377100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.377073 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:39.749774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:39.749743 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:40:39.751488 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:40:39.751457 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d90e920_634d_4d2c_8613_147a606d9bce.slice/crio-1e6406c70759405d1afa852453daebf6d35a3f757e44b3a4783204bff2129849 WatchSource:0}: Error finding container 1e6406c70759405d1afa852453daebf6d35a3f757e44b3a4783204bff2129849: Status 404 returned error can't find the container with id 1e6406c70759405d1afa852453daebf6d35a3f757e44b3a4783204bff2129849 Nov 02 21:40:40.712600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:40.712551 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" event={"ID":"5d90e920-634d-4d2c-8613-147a606d9bce","Type":"ContainerStarted","Data":"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea"} Nov 02 21:40:40.713010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:40.712642 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" event={"ID":"5d90e920-634d-4d2c-8613-147a606d9bce","Type":"ContainerStarted","Data":"1e6406c70759405d1afa852453daebf6d35a3f757e44b3a4783204bff2129849"} Nov 02 21:40:40.713010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:40.712669 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:40.828454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:40.828406 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" podStartSLOduration=1.828393401 podStartE2EDuration="1.828393401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:40.825599173 +0000 UTC m=+1880.373848954" watchObservedRunningTime="2026-11-02 21:40:40.828393401 +0000 UTC m=+1880.376643243" Nov 02 21:40:43.402246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.402210 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:40:43.406197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.406178 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.519471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.519336 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-387f88-dockercfg-zflqd\"" Nov 02 21:40:43.522989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.519988 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-387f88\"" Nov 02 21:40:43.525024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.524067 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:40:43.566000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.565913 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbmv\" (UniqueName: \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.566347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.566297 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.667522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.667438 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbmv\" (UniqueName: \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.667705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.667523 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.668470 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.668443 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:43.728745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:43.728712 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbmv\" (UniqueName: \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\") pod \"payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:44.019355 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.019287 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:44.395997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.395967 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:40:44.397097 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:40:44.397073 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fdb767_f72a_4d09_9d77_8b319d03f6e2.slice/crio-7d6a2ff5e521a94206eef7a9232257256644e0a01b7fac7ab60dac0ae87102f7 WatchSource:0}: Error finding container 7d6a2ff5e521a94206eef7a9232257256644e0a01b7fac7ab60dac0ae87102f7: Status 404 returned error can't find the container with id 7d6a2ff5e521a94206eef7a9232257256644e0a01b7fac7ab60dac0ae87102f7 Nov 02 21:40:44.739709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.739622 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" event={"ID":"e0fdb767-f72a-4d09-9d77-8b319d03f6e2","Type":"ContainerStarted","Data":"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb"} Nov 02 21:40:44.739709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.739686 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" event={"ID":"e0fdb767-f72a-4d09-9d77-8b319d03f6e2","Type":"ContainerStarted","Data":"7d6a2ff5e521a94206eef7a9232257256644e0a01b7fac7ab60dac0ae87102f7"} Nov 02 21:40:44.740107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.739827 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:44.858969 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:44.858908 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" podStartSLOduration=1.858893922 podStartE2EDuration="1.858893922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:44.856673485 +0000 UTC m=+1884.404923266" watchObservedRunningTime="2026-11-02 21:40:44.858893922 +0000 UTC m=+1884.407143702" Nov 02 21:40:49.564392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.564361 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:40:49.568644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.568623 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.666841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.666810 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.666854 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.666980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667020 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chshn\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667156 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667119 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667240 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667227 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667252 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.667462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.667448 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.701826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.701791 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:40:49.768204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768179 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768210 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768231 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768256 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768289 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768312 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chshn\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768355 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768430 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768526 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768615 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768675 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768735 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.768984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.768943 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.769140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.769116 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.770918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.770888 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.771118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.771098 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.828439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.828415 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.828566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.828471 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chshn\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-w8bqz\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:49.881686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:49.881657 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:50.312964 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.312919 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:40:50.313414 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:40:50.313390 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3504462b_dfd0_4cc7_ba4b_952574c2be7a.slice/crio-2eda10484f77d3d32785ca02d02813a7322168d925fe708c5065df52bf1b5407 WatchSource:0}: Error finding container 2eda10484f77d3d32785ca02d02813a7322168d925fe708c5065df52bf1b5407: Status 404 returned error can't find the container with id 2eda10484f77d3d32785ca02d02813a7322168d925fe708c5065df52bf1b5407 Nov 02 21:40:50.316054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.316021 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:50.316152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.316088 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:50.316152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.316128 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:50.787972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.786853 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" event={"ID":"3504462b-dfd0-4cc7-ba4b-952574c2be7a","Type":"ContainerStarted","Data":"3489bbefb52c9563693ac873330b35d9ca2798bc9fb47bb12840ec33c9e498f1"} Nov 02 21:40:50.787972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.786923 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" event={"ID":"3504462b-dfd0-4cc7-ba4b-952574c2be7a","Type":"ContainerStarted","Data":"2eda10484f77d3d32785ca02d02813a7322168d925fe708c5065df52bf1b5407"} Nov 02 21:40:50.881743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.881711 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:50.908977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:50.906457 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podStartSLOduration=1.906436091 podStartE2EDuration="1.906436091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:50.904841806 +0000 UTC m=+1890.453091587" watchObservedRunningTime="2026-11-02 21:40:50.906436091 +0000 UTC m=+1890.454685872" Nov 02 21:40:51.733068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:51.733036 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:40:51.883677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:51.883640 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:51.884117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:51.883716 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:52.882263 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:52.882226 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:52.882438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:52.882291 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:53.882247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:53.882217 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:53.882617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:53.882275 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:54.882925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:54.882895 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:54.883394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:54.882972 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:55.750251 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:55.750217 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:40:55.882484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:55.882448 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:55.882645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:55.882521 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:55.961820 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:40:55.961788 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:40:56.882425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:56.882394 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-w8bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 21:40:56.882600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:56.882452 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 21:40:58.501967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:58.501912 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:58.502488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:58.502403 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:58.503296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:58.503277 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:40:59.517533 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:59.517493 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:40:59.518035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:40:59.517749 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" containerID="cri-o://15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95" gracePeriod=30 Nov 02 21:41:04.807189 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.807159 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:41:04.894405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894368 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894405 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894408 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894437 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894574 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894618 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894677 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894700 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894746 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.894824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894765 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:04.894824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894809 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\") pod \"a95687e4-31e2-4b33-9d00-582688e3fa0e\" (UID: \"a95687e4-31e2-4b33-9d00-582688e3fa0e\") " Nov 02 21:41:04.895121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.894847 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:04.895121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895039 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:04.895224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895116 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:04.895289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895269 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:04.895568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895542 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.895703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895570 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.895703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895586 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.895703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895600 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95687e4-31e2-4b33-9d00-582688e3fa0e-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.895703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.895614 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.897001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.896975 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:04.897148 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.897011 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:04.897210 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.897167 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "kube-api-access-bddcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:04.897262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.897237 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy" pod "a95687e4-31e2-4b33-9d00-582688e3fa0e" (UID: "a95687e4-31e2-4b33-9d00-582688e3fa0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:04.904529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.904503 5859 generic.go:356] "Generic (PLEG): container finished" podID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerID="15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95" exitCode=0 Nov 02 21:41:04.904645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.904542 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" Nov 02 21:41:04.904645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.904576 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" event={"ID":"a95687e4-31e2-4b33-9d00-582688e3fa0e","Type":"ContainerDied","Data":"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95"} Nov 02 21:41:04.904645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.904635 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4" event={"ID":"a95687e4-31e2-4b33-9d00-582688e3fa0e","Type":"ContainerDied","Data":"f2a2c315af903f94fe628b71696ec7436f3445c542035ea2cdee9d73cf9f426e"} Nov 02 21:41:04.904762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.904663 5859 scope.go:122] "RemoveContainer" containerID="15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95" Nov 02 21:41:04.917144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.917113 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:41:04.918710 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.918683 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" Nov 02 21:41:04.918859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.918845 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" Nov 02 21:41:04.919259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.919242 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" containerName="istio-proxy" Nov 02 21:41:04.925319 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.924620 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:04.930824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.930803 5859 scope.go:122] "RemoveContainer" containerID="15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95" Nov 02 21:41:04.931178 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:04.931156 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95\": container with ID starting with 15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95 not found: ID does not exist" containerID="15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95" Nov 02 21:41:04.931232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.931187 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95"} err="failed to get container status \"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95\": rpc error: code = NotFound desc = could not find container \"15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95\": container with ID starting with 15eaf267bf69bf9f1a032c2a1490de8339cb4ee8e37594d97f0218e6e75b2b95 not found: ID does not exist" Nov 02 21:41:04.956396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.956358 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:41:04.996829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.996790 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:04.997284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997242 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:04.997436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997342 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:04.997436 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997374 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5htn\" (UniqueName: \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:04.997574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997553 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.997629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997577 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddcd\" (UniqueName: \"kubernetes.io/projected/a95687e4-31e2-4b33-9d00-582688e3fa0e-kube-api-access-bddcd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.997629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997588 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:04.997629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:04.997599 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95687e4-31e2-4b33-9d00-582688e3fa0e-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:05.002068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.002040 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6a9abc-serving-cert\"" Nov 02 21:41:05.098574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.098538 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.098776 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.098700 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.098776 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.098745 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5htn\" (UniqueName: \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.098915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.098854 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.099088 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:05.099049 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-serving-cert: secret "maas-api-e2e-ipp-b-6a9abc-serving-cert" not found Nov 02 21:41:05.099186 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:05.099150 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls podName:da3d4288-e693-4feb-94be-52099a171618 nodeName:}" failed. No retries permitted until 2026-11-02 21:41:05.59912806 +0000 UTC m=+1905.147377825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls") pod "maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" (UID: "da3d4288-e693-4feb-94be-52099a171618") : secret "maas-api-e2e-ipp-b-6a9abc-serving-cert" not found Nov 02 21:41:05.099501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.099480 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.101253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.101234 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.162758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.162719 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5htn\" (UniqueName: \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.207731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.207700 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:41:05.210656 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.210628 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-q58d4"] Nov 02 21:41:05.611107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.611065 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.613534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.613512 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:05.840848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:05.840810 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:06.217898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:06.217869 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:41:06.218236 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:41:06.218213 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3d4288_e693_4feb_94be_52099a171618.slice/crio-547014f71441e2f1f282f6f7fc9f169219834f8de73eedadb197a7d40f080a40 WatchSource:0}: Error finding container 547014f71441e2f1f282f6f7fc9f169219834f8de73eedadb197a7d40f080a40: Status 404 returned error can't find the container with id 547014f71441e2f1f282f6f7fc9f169219834f8de73eedadb197a7d40f080a40 Nov 02 21:41:06.573198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:06.573158 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95687e4-31e2-4b33-9d00-582688e3fa0e" path="/var/lib/kubelet/pods/a95687e4-31e2-4b33-9d00-582688e3fa0e/volumes" Nov 02 21:41:06.927905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:06.927822 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" event={"ID":"da3d4288-e693-4feb-94be-52099a171618","Type":"ContainerStarted","Data":"547014f71441e2f1f282f6f7fc9f169219834f8de73eedadb197a7d40f080a40"} Nov 02 21:41:07.935828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:07.935790 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" event={"ID":"da3d4288-e693-4feb-94be-52099a171618","Type":"ContainerStarted","Data":"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6"} Nov 02 21:41:07.936256 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:07.935883 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:08.053873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:08.052841 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" podStartSLOduration=3.185394055 podStartE2EDuration="4.052821433s" totalImagesPullingTime="867.427378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:08.050118111 +0000 UTC m=+1907.598367945" watchObservedRunningTime="2026-11-02 21:41:08.052821433 +0000 UTC m=+1907.601071209" Nov 02 21:41:13.951934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:13.951906 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:41:25.945329 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:25.945288 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:41:41.010369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.010339 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:41:41.016173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.016153 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.066755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.066729 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.066904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.066842 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.067000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.066983 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.067061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.067014 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzk8h\" (UniqueName: \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.068993 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.068970 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:41:41.115575 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.115553 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ce2453-serving-cert\"" Nov 02 21:41:41.168534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.168506 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.168690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.168616 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.168690 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:41.168648 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-serving-cert: secret "maas-api-e2e-ipp-a-ce2453-serving-cert" not found Nov 02 21:41:41.168817 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:41.168701 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls podName:d69aec57-2fc4-4ca6-8c89-2e1923d64e4c nodeName:}" failed. No retries permitted until 2026-11-02 21:41:41.66868581 +0000 UTC m=+1941.216935576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls") pod "maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" (UID: "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c") : secret "maas-api-e2e-ipp-a-ce2453-serving-cert" not found Nov 02 21:41:41.168817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.168735 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.168817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.168762 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzk8h\" (UniqueName: \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.169490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.169466 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.171152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.171130 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.228089 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.228066 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzk8h\" (UniqueName: \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.674150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.674114 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.676783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.676756 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:41.930411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:41.930335 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:42.311902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.311869 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:41:42.313494 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:41:42.313469 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69aec57_2fc4_4ca6_8c89_2e1923d64e4c.slice/crio-7f51f8e166738023936e40d3550a65714f29299f57e0d449f72556a79c1d3a46 WatchSource:0}: Error finding container 7f51f8e166738023936e40d3550a65714f29299f57e0d449f72556a79c1d3a46: Status 404 returned error can't find the container with id 7f51f8e166738023936e40d3550a65714f29299f57e0d449f72556a79c1d3a46 Nov 02 21:41:42.315100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.315079 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:41:42.636054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.636026 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:41:42.636215 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.636191 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" containerID="cri-o://0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f" gracePeriod=30 Nov 02 21:41:42.686570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.686543 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:41:42.686782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:42.686755 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" containerID="cri-o://d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977" gracePeriod=30 Nov 02 21:41:43.163353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:43.163318 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" event={"ID":"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c","Type":"ContainerStarted","Data":"7f51f8e166738023936e40d3550a65714f29299f57e0d449f72556a79c1d3a46"} Nov 02 21:41:44.048559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:44.048521 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:41:44.048932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:44.048584 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:41:44.197902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:44.197819 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" event={"ID":"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c","Type":"ContainerStarted","Data":"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6"} Nov 02 21:41:44.315656 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:44.315547 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" podStartSLOduration=3.366304903 podStartE2EDuration="4.31553266s" totalImagesPullingTime="949.227757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:44.312994877 +0000 UTC m=+1943.861244655" watchObservedRunningTime="2026-11-02 21:41:44.31553266 +0000 UTC m=+1943.863782439" Nov 02 21:41:45.205107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:45.205080 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:47.953522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:47.953494 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:41:47.977851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:47.977828 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:41:48.062551 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062523 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.062677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062565 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.062677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062601 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.062677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062638 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.062677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062667 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.062898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062737 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.062898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062797 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.062898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062825 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.062898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062853 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.062898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062860 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062873 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062901 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062984 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063045 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063087 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.063174 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063131 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063211 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063239 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\") pod \"a5ab1639-650d-41fa-8f7b-366327d5de34\" (UID: \"a5ab1639-650d-41fa-8f7b-366327d5de34\") " Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.062928 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063301 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063007 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063052 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063275 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:48.063472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063428 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\") pod \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\" (UID: \"3e2a5e75-9891-41f5-916a-a837f2f73f4d\") " Nov 02 21:41:48.063879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063511 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.063879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063825 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:48.063879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.063868 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.064144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064098 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064113 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.064144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064128 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064144 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ab1639-650d-41fa-8f7b-366327d5de34-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064159 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064172 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064185 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064201 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064215 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.064386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.064228 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.065230 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.065161 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.065619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.065595 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:48.065709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.065686 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "kube-api-access-ppvvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:48.065889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.065871 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "kube-api-access-hntzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:48.066088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.066069 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:48.066246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.066225 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:48.066304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.066262 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo" pod "3e2a5e75-9891-41f5-916a-a837f2f73f4d" (UID: "3e2a5e75-9891-41f5-916a-a837f2f73f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:48.066882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.066866 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo" pod "a5ab1639-650d-41fa-8f7b-366327d5de34" (UID: "a5ab1639-650d-41fa-8f7b-366327d5de34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165096 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165118 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165127 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165135 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165144 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165152 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165155 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165161 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppvvd\" (UniqueName: \"kubernetes.io/projected/a5ab1639-650d-41fa-8f7b-366327d5de34-kube-api-access-ppvvd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165444 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165171 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hntzr\" (UniqueName: \"kubernetes.io/projected/3e2a5e75-9891-41f5-916a-a837f2f73f4d-kube-api-access-hntzr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.165444 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.165179 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2a5e75-9891-41f5-916a-a837f2f73f4d-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:48.224905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.224877 5859 generic.go:356] "Generic (PLEG): container finished" podID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerID="d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977" exitCode=0 Nov 02 21:41:48.225040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.224928 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" Nov 02 21:41:48.225040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.224978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" event={"ID":"3e2a5e75-9891-41f5-916a-a837f2f73f4d","Type":"ContainerDied","Data":"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977"} Nov 02 21:41:48.225040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.225025 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb" event={"ID":"3e2a5e75-9891-41f5-916a-a837f2f73f4d","Type":"ContainerDied","Data":"d3591367c453f5c45ab18511aa10764ab830242f16ba0df3f28670df9c6e514d"} Nov 02 21:41:48.225180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.225046 5859 scope.go:122] "RemoveContainer" containerID="d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977" Nov 02 21:41:48.226437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.226419 5859 generic.go:356] "Generic (PLEG): container finished" podID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerID="0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f" exitCode=0 Nov 02 21:41:48.226525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.226447 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" event={"ID":"a5ab1639-650d-41fa-8f7b-366327d5de34","Type":"ContainerDied","Data":"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f"} Nov 02 21:41:48.226525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.226475 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" event={"ID":"a5ab1639-650d-41fa-8f7b-366327d5de34","Type":"ContainerDied","Data":"715d3ebaf7db0d1069724adf2b705bdd929da877860baa042d322cbe3f36ef01"} Nov 02 21:41:48.226525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.226483 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f" Nov 02 21:41:48.238385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.238189 5859 scope.go:122] "RemoveContainer" containerID="d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977" Nov 02 21:41:48.242116 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:48.242088 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977\": container with ID starting with d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977 not found: ID does not exist" containerID="d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977" Nov 02 21:41:48.242225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.242124 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977"} err="failed to get container status \"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977\": rpc error: code = NotFound desc = could not find container \"d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977\": container with ID starting with d4601c63b91f55c2d977b6a2931e6b4610d457926ea66622c810abaa01c5e977 not found: ID does not exist" Nov 02 21:41:48.242225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.242148 5859 scope.go:122] "RemoveContainer" containerID="0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f" Nov 02 21:41:48.254373 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.254350 5859 scope.go:122] "RemoveContainer" containerID="0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f" Nov 02 21:41:48.254645 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:48.254622 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f\": container with ID starting with 0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f not found: ID does not exist" containerID="0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f" Nov 02 21:41:48.254738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.254649 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f"} err="failed to get container status \"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f\": rpc error: code = NotFound desc = could not find container \"0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f\": container with ID starting with 0354bc9ab18a129610790afe4a297b69c6746338e8ebd302bbfb7e65e05e115f not found: ID does not exist" Nov 02 21:41:48.458531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.458464 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:41:48.461196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.461175 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-tzk5f"] Nov 02 21:41:48.584218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.584183 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" path="/var/lib/kubelet/pods/a5ab1639-650d-41fa-8f7b-366327d5de34/volumes" Nov 02 21:41:48.584485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.584472 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:41:48.584528 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:48.584500 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-z8twb"] Nov 02 21:41:50.572680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:50.572648 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" path="/var/lib/kubelet/pods/3e2a5e75-9891-41f5-916a-a837f2f73f4d/volumes" Nov 02 21:41:51.187310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187277 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:41:51.187854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187841 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" Nov 02 21:41:51.187900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187855 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" Nov 02 21:41:51.187900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187877 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" Nov 02 21:41:51.187900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187882 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" Nov 02 21:41:51.188020 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.187993 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2a5e75-9891-41f5-916a-a837f2f73f4d" containerName="istio-proxy" Nov 02 21:41:51.188020 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.188007 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ab1639-650d-41fa-8f7b-366327d5de34" containerName="istio-proxy" Nov 02 21:41:51.193282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.193263 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.216905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.216886 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:41:51.254701 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.254680 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:41:51.296983 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.296929 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.297121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.297037 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl8m\" (UniqueName: \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.297121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.297100 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.297349 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.297333 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.301561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.301543 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-387f88-serving-cert\"" Nov 02 21:41:51.398822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.398795 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl8m\" (UniqueName: \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.399040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.398836 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.399126 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:51.399055 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-serving-cert: secret "maas-api-e2e-ipp-b-387f88-serving-cert" not found Nov 02 21:41:51.399126 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:51.399117 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls podName:f8c10886-3e07-418d-9954-c5a0c79cb359 nodeName:}" failed. No retries permitted until 2026-11-02 21:41:51.899102756 +0000 UTC m=+1951.447352532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls") pod "maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" (UID: "f8c10886-3e07-418d-9954-c5a0c79cb359") : secret "maas-api-e2e-ipp-b-387f88-serving-cert" not found Nov 02 21:41:51.399254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.399060 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.399254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.399215 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.399786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.399759 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.401434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.401413 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.458615 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.458499 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl8m\" (UniqueName: \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.906714 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.906687 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:51.909125 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:51.909102 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") pod \"maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:52.107489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:52.107462 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:52.483377 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:52.483351 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:41:52.484695 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:41:52.484659 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c10886_3e07_418d_9954_c5a0c79cb359.slice/crio-9af216d34f97bdbc914247935ffecfa232d8462a394922d8319fd98953cadfd9 WatchSource:0}: Error finding container 9af216d34f97bdbc914247935ffecfa232d8462a394922d8319fd98953cadfd9: Status 404 returned error can't find the container with id 9af216d34f97bdbc914247935ffecfa232d8462a394922d8319fd98953cadfd9 Nov 02 21:41:53.273534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:53.273504 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" event={"ID":"f8c10886-3e07-418d-9954-c5a0c79cb359","Type":"ContainerStarted","Data":"9af216d34f97bdbc914247935ffecfa232d8462a394922d8319fd98953cadfd9"} Nov 02 21:41:54.281686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:54.281649 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" event={"ID":"f8c10886-3e07-418d-9954-c5a0c79cb359","Type":"ContainerStarted","Data":"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13"} Nov 02 21:41:54.282086 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:54.281742 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:41:54.398312 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:41:54.398254 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" podStartSLOduration=2.5846033569999998 podStartE2EDuration="3.398241263s" totalImagesPullingTime="813.637906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:54.395644733 +0000 UTC m=+1953.943894533" watchObservedRunningTime="2026-11-02 21:41:54.398241263 +0000 UTC m=+1953.946491044" Nov 02 21:41:55.945892 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:41:55.945851 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:42:00.293460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:00.293384 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:42:03.695899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.695871 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:42:03.700420 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.700403 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.741306 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.741275 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.741411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.741327 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xq6\" (UniqueName: \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.741454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.741415 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.741503 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.741473 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.754604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.754578 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:42:03.801003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.800984 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3e3997-serving-cert\"" Nov 02 21:42:03.843091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.843064 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.843226 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:42:03.843209 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-serving-cert: secret "maas-api-e2e-ipp-b-3e3997-serving-cert" not found Nov 02 21:42:03.843276 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.843235 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.843323 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:42:03.843282 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls podName:984c32f6-6968-45db-bfac-a92667868c08 nodeName:}" failed. No retries permitted until 2026-11-02 21:42:04.343263258 +0000 UTC m=+1963.891513020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls") pod "maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" (UID: "984c32f6-6968-45db-bfac-a92667868c08") : secret "maas-api-e2e-ipp-b-3e3997-serving-cert" not found Nov 02 21:42:03.843323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.843316 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xq6\" (UniqueName: \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.843425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.843363 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.844121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.844082 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.845681 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.845661 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:03.900421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:03.900393 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xq6\" (UniqueName: \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:04.357520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:04.357485 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:04.360001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:04.359978 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:04.614116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:04.614041 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:04.988234 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:04.988207 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:42:04.989985 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:42:04.989946 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984c32f6_6968_45db_bfac_a92667868c08.slice/crio-675219e7c6fd826a8fddbc11948e240cb77d9b173ef1d845145bd583c80517bd WatchSource:0}: Error finding container 675219e7c6fd826a8fddbc11948e240cb77d9b173ef1d845145bd583c80517bd: Status 404 returned error can't find the container with id 675219e7c6fd826a8fddbc11948e240cb77d9b173ef1d845145bd583c80517bd Nov 02 21:42:05.360153 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:05.360122 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" event={"ID":"984c32f6-6968-45db-bfac-a92667868c08","Type":"ContainerStarted","Data":"675219e7c6fd826a8fddbc11948e240cb77d9b173ef1d845145bd583c80517bd"} Nov 02 21:42:06.368374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:06.368336 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" event={"ID":"984c32f6-6968-45db-bfac-a92667868c08","Type":"ContainerStarted","Data":"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f"} Nov 02 21:42:06.368771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:06.368402 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:12.381585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:12.381505 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:42:12.497068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:12.497002 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" podStartSLOduration=8.616206306 podStartE2EDuration="9.496989822s" totalImagesPullingTime="880.783516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:42:06.490176646 +0000 UTC m=+1966.038426479" watchObservedRunningTime="2026-11-02 21:42:12.496989822 +0000 UTC m=+1972.045239602" Nov 02 21:42:14.421011 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.420985 5859 generic.go:356] "Generic (PLEG): container finished" podID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerID="363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd" exitCode=0 Nov 02 21:42:14.421381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.421042 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerDied","Data":"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd"} Nov 02 21:42:14.421381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.421371 5859 scope.go:122] "RemoveContainer" containerID="363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd" Nov 02 21:42:14.422795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.422766 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:14.422887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.422834 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:14.422887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.422865 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:14.628112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:14.628075 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:42:15.436616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:15.436581 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerStarted","Data":"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79"} Nov 02 21:42:15.450118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:15.450094 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:42:15.451252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:15.451228 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:42:15.451363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:15.451272 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 21:42:16.452111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:16.452070 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:42:16.452613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:16.452232 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 21:42:17.449790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:17.449759 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:42:17.450002 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:17.449822 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 21:42:18.450347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:18.450311 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:42:18.450767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:18.450381 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 21:42:19.449933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:19.449899 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 21:42:19.450135 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:19.449977 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 21:42:20.454314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:20.454284 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:42:20.454735 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:20.454667 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:42:20.455224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:20.455204 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:42:25.943824 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:42:25.943793 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:42:38.384449 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:38.384373 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:42:38.384941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:38.384617 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" containerID="cri-o://3f476694ad1e887d16eddb5d4f37c338fa23bf46e292a8a229f35f9063e8355a" gracePeriod=30 Nov 02 21:42:41.566236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:41.566199 5859 patch_prober.go:29] interesting pod/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:42:41.566801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:41.566769 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:42:42.628278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.628224 5859 generic.go:356] "Generic (PLEG): container finished" podID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerID="53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24" exitCode=0 Nov 02 21:42:42.628635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.628293 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerDied","Data":"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24"} Nov 02 21:42:42.628635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.628581 5859 scope.go:122] "RemoveContainer" containerID="53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24" Nov 02 21:42:42.629830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.629805 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:42.629882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.629868 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:42.629916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:42.629899 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:42:43.647980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.647575 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerStarted","Data":"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd"} Nov 02 21:42:43.650728 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.650648 5859 generic.go:356] "Generic (PLEG): container finished" podID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerID="3f476694ad1e887d16eddb5d4f37c338fa23bf46e292a8a229f35f9063e8355a" exitCode=0 Nov 02 21:42:43.650728 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.650696 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerDied","Data":"3f476694ad1e887d16eddb5d4f37c338fa23bf46e292a8a229f35f9063e8355a"} Nov 02 21:42:43.651000 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.650751 5859 scope.go:122] "RemoveContainer" containerID="21d1d32b18611d5ae53df600c9199c5639938bc07cabb58cbcd0f715442465a0" Nov 02 21:42:43.731355 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.731318 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:42:43.830541 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830502 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.830738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830571 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.830738 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830727 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.830869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830798 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.830869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830853 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:42:43.831024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.830858 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.831024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831003 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.831143 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831038 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.831196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831160 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.831196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831174 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:42:43.831298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831199 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\") pod \"58597b2e-a889-49dd-9f2c-b039f4251bbd\" (UID: \"58597b2e-a889-49dd-9f2c-b039f4251bbd\") " Nov 02 21:42:43.831410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831387 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:42:43.831500 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831471 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:42:43.831624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831603 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:42:43.831922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831901 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.831922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831924 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.832111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831937 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.832111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.831989 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.832111 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.832003 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58597b2e-a889-49dd-9f2c-b039f4251bbd-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.833705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.833674 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:42:43.833847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.833825 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:42:43.833909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.833848 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "kube-api-access-cvwrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:42:43.834007 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.833920 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo" pod "58597b2e-a889-49dd-9f2c-b039f4251bbd" (UID: "58597b2e-a889-49dd-9f2c-b039f4251bbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:42:43.933787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.933699 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.933787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.933736 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvwrn\" (UniqueName: \"kubernetes.io/projected/58597b2e-a889-49dd-9f2c-b039f4251bbd-kube-api-access-cvwrn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.933787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.933752 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:43.933787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:43.933766 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58597b2e-a889-49dd-9f2c-b039f4251bbd-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:44.528976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.526106 5859 patch_prober.go:29] interesting pod/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 21:42:44.528976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.526165 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 21:42:44.528976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.526480 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:42:44.661856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.661823 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" Nov 02 21:42:44.661856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.661846 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6" event={"ID":"58597b2e-a889-49dd-9f2c-b039f4251bbd","Type":"ContainerDied","Data":"f988206c4135e0d3a7e329ef198dc0cb03d2fa293dd0473631c862e376368819"} Nov 02 21:42:44.662393 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.661910 5859 scope.go:122] "RemoveContainer" containerID="3f476694ad1e887d16eddb5d4f37c338fa23bf46e292a8a229f35f9063e8355a" Nov 02 21:42:44.844770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.844738 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:42:44.894101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:44.894075 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-54e844-openshift-default-85f66bbf99-7l7x6"] Nov 02 21:42:45.523985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:45.523338 5859 patch_prober.go:29] interesting pod/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 21:42:45.523985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:45.523415 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 21:42:46.529988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:46.528827 5859 patch_prober.go:29] interesting pod/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 21:42:46.529988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:46.528885 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 21:42:46.571422 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:46.571394 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" path="/var/lib/kubelet/pods/58597b2e-a889-49dd-9f2c-b039f4251bbd/volumes" Nov 02 21:42:46.646006 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:46.645978 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:42:47.530974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:47.524357 5859 patch_prober.go:29] interesting pod/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 21:42:47.530974 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:47.524417 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 21:42:48.530619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:48.530589 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:42:48.532144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:42:48.532118 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:42:55.945442 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:42:55.945406 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:43:10.082329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:10.082291 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:43:15.083221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.083168 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" podUID="5d90e920-634d-4d2c-8613-147a606d9bce" containerName="payload-processing" containerID="cri-o://492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea" gracePeriod=25 Nov 02 21:43:15.346033 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.345981 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:43:15.396526 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.396502 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\") pod \"5d90e920-634d-4d2c-8613-147a606d9bce\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " Nov 02 21:43:15.396667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.396549 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\" (UniqueName: \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\") pod \"5d90e920-634d-4d2c-8613-147a606d9bce\" (UID: \"5d90e920-634d-4d2c-8613-147a606d9bce\") " Nov 02 21:43:15.396791 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.396769 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume" pod "5d90e920-634d-4d2c-8613-147a606d9bce" (UID: "5d90e920-634d-4d2c-8613-147a606d9bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:15.397046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.397030 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d90e920-634d-4d2c-8613-147a606d9bce-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:15.398452 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.398433 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z" pod "5d90e920-634d-4d2c-8613-147a606d9bce" (UID: "5d90e920-634d-4d2c-8613-147a606d9bce"). InnerVolumeSpecName "kube-api-access-sjv4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:15.498253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.498227 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjv4z\" (UniqueName: \"kubernetes.io/projected/5d90e920-634d-4d2c-8613-147a606d9bce-kube-api-access-sjv4z\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:15.878818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.878783 5859 generic.go:356] "Generic (PLEG): container finished" podID="5d90e920-634d-4d2c-8613-147a606d9bce" containerID="492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea" exitCode=0 Nov 02 21:43:15.879015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.878828 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" Nov 02 21:43:15.879015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.878865 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" event={"ID":"5d90e920-634d-4d2c-8613-147a606d9bce","Type":"ContainerDied","Data":"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea"} Nov 02 21:43:15.879015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.878919 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb" event={"ID":"5d90e920-634d-4d2c-8613-147a606d9bce","Type":"ContainerDied","Data":"1e6406c70759405d1afa852453daebf6d35a3f757e44b3a4783204bff2129849"} Nov 02 21:43:15.879015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.878945 5859 scope.go:122] "RemoveContainer" containerID="492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea" Nov 02 21:43:15.889384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.889368 5859 scope.go:122] "RemoveContainer" containerID="492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea" Nov 02 21:43:15.889637 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:43:15.889616 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea\": container with ID starting with 492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea not found: ID does not exist" containerID="492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea" Nov 02 21:43:15.889709 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:15.889643 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea"} err="failed to get container status \"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea\": rpc error: code = NotFound desc = could not find container \"492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea\": container with ID starting with 492d7ef4efb2b269d82e34d13c85448745acfb554864639fc37e9400d80ba1ea not found: ID does not exist" Nov 02 21:43:16.043957 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:16.043913 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:43:16.093645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:16.093612 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ce2453-5d6d486657-7dbwb"] Nov 02 21:43:16.570460 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:16.570433 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d90e920-634d-4d2c-8613-147a606d9bce" path="/var/lib/kubelet/pods/5d90e920-634d-4d2c-8613-147a606d9bce/volumes" Nov 02 21:43:17.304359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:17.304328 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:43:22.304841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.304806 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" podUID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" containerName="payload-pre-processing" containerID="cri-o://933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb" gracePeriod=25 Nov 02 21:43:22.570737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.570710 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:43:22.674842 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.674815 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\") pod \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " Nov 02 21:43:22.675025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.674866 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\" (UniqueName: \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\") pod \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\" (UID: \"e0fdb767-f72a-4d09-9d77-8b319d03f6e2\") " Nov 02 21:43:22.675192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.675163 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume" pod "e0fdb767-f72a-4d09-9d77-8b319d03f6e2" (UID: "e0fdb767-f72a-4d09-9d77-8b319d03f6e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:22.675445 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.675422 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:22.676855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.676831 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv" pod "e0fdb767-f72a-4d09-9d77-8b319d03f6e2" (UID: "e0fdb767-f72a-4d09-9d77-8b319d03f6e2"). InnerVolumeSpecName "kube-api-access-9qbmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:22.776739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.776713 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbmv\" (UniqueName: \"kubernetes.io/projected/e0fdb767-f72a-4d09-9d77-8b319d03f6e2-kube-api-access-9qbmv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:22.923660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.923580 5859 generic.go:356] "Generic (PLEG): container finished" podID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" containerID="933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb" exitCode=0 Nov 02 21:43:22.923660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.923643 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" Nov 02 21:43:22.923909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.923668 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" event={"ID":"e0fdb767-f72a-4d09-9d77-8b319d03f6e2","Type":"ContainerDied","Data":"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb"} Nov 02 21:43:22.923909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.923734 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn" event={"ID":"e0fdb767-f72a-4d09-9d77-8b319d03f6e2","Type":"ContainerDied","Data":"7d6a2ff5e521a94206eef7a9232257256644e0a01b7fac7ab60dac0ae87102f7"} Nov 02 21:43:22.923909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.923765 5859 scope.go:122] "RemoveContainer" containerID="933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb" Nov 02 21:43:22.934441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.934420 5859 scope.go:122] "RemoveContainer" containerID="933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb" Nov 02 21:43:22.934699 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:43:22.934679 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb\": container with ID starting with 933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb not found: ID does not exist" containerID="933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb" Nov 02 21:43:22.934749 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:22.934706 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb"} err="failed to get container status \"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb\": rpc error: code = NotFound desc = could not find container \"933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb\": container with ID starting with 933b29d97c8cb5dceaf189b243c66e11c701886fecda6d5796483e9bfdb44adb not found: ID does not exist" Nov 02 21:43:23.093818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:23.093701 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:43:23.141894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:23.141865 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-387f88-7f56f67795-r5chn"] Nov 02 21:43:24.571619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:43:24.571585 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" path="/var/lib/kubelet/pods/e0fdb767-f72a-4d09-9d77-8b319d03f6e2/volumes" Nov 02 21:43:25.945442 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:43:25.945404 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:43:55.947889 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:43:55.947849 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:44:25.943945 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:44:25.943903 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:44:47.527687 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.527657 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:44:47.528307 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528293 5859 cpu_manager.go:414] "containerMap: removing container" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:44:47.528307 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528308 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528327 5859 cpu_manager.go:414] "containerMap: removing container" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528332 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528343 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5d90e920-634d-4d2c-8613-147a606d9bce" containerName="payload-processing" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528348 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d90e920-634d-4d2c-8613-147a606d9bce" containerName="payload-processing" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528366 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" containerName="payload-pre-processing" Nov 02 21:44:47.528398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528370 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" containerName="payload-pre-processing" Nov 02 21:44:47.528571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528454 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:44:47.528571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528466 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d90e920-634d-4d2c-8613-147a606d9bce" containerName="payload-processing" Nov 02 21:44:47.528571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.528474 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fdb767-f72a-4d09-9d77-8b319d03f6e2" containerName="payload-pre-processing" Nov 02 21:44:47.532232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.532212 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.597754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.597713 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:44:47.642286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.642252 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa60bc\"" Nov 02 21:44:47.649979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.642630 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa60bc-dockercfg-7djms\"" Nov 02 21:44:47.682417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.682375 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.683075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.682661 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf22h\" (UniqueName: \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.784451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.784375 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.784451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.784411 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf22h\" (UniqueName: \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.785098 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.785074 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.842198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.842179 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf22h\" (UniqueName: \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\") pod \"payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:47.848358 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:47.848341 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:48.219571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:48.219548 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:44:48.221349 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:44:48.221315 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2653f2d_c710_4cbf_b119_1a1237e3c523.slice/crio-4195eaadb806fc4dc1c98f206ba566c54a1dabf55957c565490685e454836957 WatchSource:0}: Error finding container 4195eaadb806fc4dc1c98f206ba566c54a1dabf55957c565490685e454836957: Status 404 returned error can't find the container with id 4195eaadb806fc4dc1c98f206ba566c54a1dabf55957c565490685e454836957 Nov 02 21:44:48.464041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:48.464006 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" event={"ID":"a2653f2d-c710-4cbf-b119-1a1237e3c523","Type":"ContainerStarted","Data":"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc"} Nov 02 21:44:48.464209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:48.464067 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" event={"ID":"a2653f2d-c710-4cbf-b119-1a1237e3c523","Type":"ContainerStarted","Data":"4195eaadb806fc4dc1c98f206ba566c54a1dabf55957c565490685e454836957"} Nov 02 21:44:48.464277 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:48.464231 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:44:48.581736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:48.581690 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" podStartSLOduration=1.581677306 podStartE2EDuration="1.581677306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:48.578683823 +0000 UTC m=+2128.126933603" watchObservedRunningTime="2026-11-02 21:44:48.581677306 +0000 UTC m=+2128.129927087" Nov 02 21:44:55.946263 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:44:55.946222 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:44:59.474439 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:44:59.474412 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:45:00.208902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.208829 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:00.209517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.209502 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="58597b2e-a889-49dd-9f2c-b039f4251bbd" containerName="istio-proxy" Nov 02 21:45:00.213014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.212998 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.315927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.315894 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:00.322330 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.322307 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 21:45:00.329662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.329634 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x447p\" (UniqueName: \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.329781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.329690 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.430693 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.430661 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x447p\" (UniqueName: \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.430860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.430714 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.431563 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.431532 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.487270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.487219 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x447p\" (UniqueName: \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\") pod \"maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.526162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.526141 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:00.903424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:00.903397 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:00.904634 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:45:00.904607 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b3b06_2fdc_4cbe_b7e0_b38e0055c146.slice/crio-2f5cc68688f0c8840d322888e683209c1bc78193bdc3903d31d635934b9633ac WatchSource:0}: Error finding container 2f5cc68688f0c8840d322888e683209c1bc78193bdc3903d31d635934b9633ac: Status 404 returned error can't find the container with id 2f5cc68688f0c8840d322888e683209c1bc78193bdc3903d31d635934b9633ac Nov 02 21:45:01.546297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:01.546215 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerStarted","Data":"93f0497284225f421cb7374b06c7d1672006882c02b7a2e04a6e3129b539b48f"} Nov 02 21:45:01.546297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:01.546295 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerStarted","Data":"2f5cc68688f0c8840d322888e683209c1bc78193bdc3903d31d635934b9633ac"} Nov 02 21:45:01.659882 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:01.659836 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" podStartSLOduration=1.659823673 podStartE2EDuration="1.659823673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:01.65878801 +0000 UTC m=+2141.207037801" watchObservedRunningTime="2026-11-02 21:45:01.659823673 +0000 UTC m=+2141.208073454" Nov 02 21:45:02.554481 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:02.554445 5859 generic.go:356] "Generic (PLEG): container finished" podID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerID="93f0497284225f421cb7374b06c7d1672006882c02b7a2e04a6e3129b539b48f" exitCode=60 Nov 02 21:45:02.554870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:02.554521 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerDied","Data":"93f0497284225f421cb7374b06c7d1672006882c02b7a2e04a6e3129b539b48f"} Nov 02 21:45:02.554870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:02.554760 5859 scope.go:122] "RemoveContainer" containerID="93f0497284225f421cb7374b06c7d1672006882c02b7a2e04a6e3129b539b48f" Nov 02 21:45:03.562869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:03.562786 5859 generic.go:356] "Generic (PLEG): container finished" podID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerID="deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f" exitCode=60 Nov 02 21:45:03.562869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:03.562843 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerDied","Data":"deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f"} Nov 02 21:45:03.563364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:03.562884 5859 scope.go:122] "RemoveContainer" containerID="93f0497284225f421cb7374b06c7d1672006882c02b7a2e04a6e3129b539b48f" Nov 02 21:45:03.563364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:03.563226 5859 scope.go:122] "RemoveContainer" containerID="deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f" Nov 02 21:45:03.563545 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:03.563526 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm_odh-ai-gateway-infra(c65b3b06-2fdc-4cbe-b7e0-b38e0055c146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" Nov 02 21:45:04.571998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:04.571968 5859 scope.go:122] "RemoveContainer" containerID="deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f" Nov 02 21:45:04.572387 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:04.572270 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm_odh-ai-gateway-infra(c65b3b06-2fdc-4cbe-b7e0-b38e0055c146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" Nov 02 21:45:07.850608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:07.850569 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:45:07.851030 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:07.850799 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" podUID="da3d4288-e693-4feb-94be-52099a171618" containerName="maas-api" containerID="cri-o://3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6" gracePeriod=30 Nov 02 21:45:08.132001 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.131973 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:45:08.234837 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.234793 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") pod \"da3d4288-e693-4feb-94be-52099a171618\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " Nov 02 21:45:08.235064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.234866 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\" (UniqueName: \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\") pod \"da3d4288-e693-4feb-94be-52099a171618\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " Nov 02 21:45:08.235064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.234917 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\") pod \"da3d4288-e693-4feb-94be-52099a171618\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " Nov 02 21:45:08.235064 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.234998 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\") pod \"da3d4288-e693-4feb-94be-52099a171618\" (UID: \"da3d4288-e693-4feb-94be-52099a171618\") " Nov 02 21:45:08.235741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.235696 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca" pod "da3d4288-e693-4feb-94be-52099a171618" (UID: "da3d4288-e693-4feb-94be-52099a171618"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:08.237758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.237731 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls" pod "da3d4288-e693-4feb-94be-52099a171618" (UID: "da3d4288-e693-4feb-94be-52099a171618"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:08.238152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.238130 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs" pod "da3d4288-e693-4feb-94be-52099a171618" (UID: "da3d4288-e693-4feb-94be-52099a171618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:08.238471 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.238450 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn" pod "da3d4288-e693-4feb-94be-52099a171618" (UID: "da3d4288-e693-4feb-94be-52099a171618"). InnerVolumeSpecName "kube-api-access-p5htn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:08.336463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.336428 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:08.336463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.336462 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5htn\" (UniqueName: \"kubernetes.io/projected/da3d4288-e693-4feb-94be-52099a171618-kube-api-access-p5htn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:08.336699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.336476 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3d4288-e693-4feb-94be-52099a171618-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:08.336699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.336490 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3d4288-e693-4feb-94be-52099a171618-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:08.596284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.596253 5859 generic.go:356] "Generic (PLEG): container finished" podID="da3d4288-e693-4feb-94be-52099a171618" containerID="3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6" exitCode=0 Nov 02 21:45:08.596440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.596299 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" Nov 02 21:45:08.596440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.596335 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" event={"ID":"da3d4288-e693-4feb-94be-52099a171618","Type":"ContainerDied","Data":"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6"} Nov 02 21:45:08.596440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.596392 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq" event={"ID":"da3d4288-e693-4feb-94be-52099a171618","Type":"ContainerDied","Data":"547014f71441e2f1f282f6f7fc9f169219834f8de73eedadb197a7d40f080a40"} Nov 02 21:45:08.596440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.596418 5859 scope.go:122] "RemoveContainer" containerID="3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6" Nov 02 21:45:08.606653 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.606631 5859 scope.go:122] "RemoveContainer" containerID="3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6" Nov 02 21:45:08.606920 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:08.606897 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6\": container with ID starting with 3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6 not found: ID does not exist" containerID="3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6" Nov 02 21:45:08.607042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.606931 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6"} err="failed to get container status \"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6\": rpc error: code = NotFound desc = could not find container \"3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6\": container with ID starting with 3b572a06dfa7d5d17e68e133c716d11503cf984399c1f7ac25c7f4dd4583f1c6 not found: ID does not exist" Nov 02 21:45:08.762663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.762631 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:45:08.770638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:08.770612 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6a9abc-64cf9cdf49-4lkgq"] Nov 02 21:45:10.579149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:10.579108 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3d4288-e693-4feb-94be-52099a171618" path="/var/lib/kubelet/pods/da3d4288-e693-4feb-94be-52099a171618/volumes" Nov 02 21:45:15.563499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:15.563470 5859 scope.go:122] "RemoveContainer" containerID="deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f" Nov 02 21:45:16.646326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:16.646278 5859 generic.go:356] "Generic (PLEG): container finished" podID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerID="56e74ad7960dba0e17278092adf7f63c5042787cd52434cf98e0d40de8e0c074" exitCode=60 Nov 02 21:45:16.646326 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:16.646322 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerDied","Data":"56e74ad7960dba0e17278092adf7f63c5042787cd52434cf98e0d40de8e0c074"} Nov 02 21:45:16.646775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:16.646363 5859 scope.go:122] "RemoveContainer" containerID="deaeac4ec6dfc45a20a58beef0f7356097307ef76325303714b133560128895f" Nov 02 21:45:16.646775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:16.646671 5859 scope.go:122] "RemoveContainer" containerID="56e74ad7960dba0e17278092adf7f63c5042787cd52434cf98e0d40de8e0c074" Nov 02 21:45:16.647003 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:16.646947 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm_odh-ai-gateway-infra(c65b3b06-2fdc-4cbe-b7e0-b38e0055c146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" Nov 02 21:45:17.774597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.774566 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:17.927637 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.927612 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:17.990392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.990368 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\") pod \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " Nov 02 21:45:17.990511 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.990434 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\" (UniqueName: \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\") pod \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\" (UID: \"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146\") " Nov 02 21:45:17.990720 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.990698 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca" pod "c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" (UID: "c65b3b06-2fdc-4cbe-b7e0-b38e0055c146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:17.990857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.990839 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:17.992352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:17.992331 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p" pod "c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" (UID: "c65b3b06-2fdc-4cbe-b7e0-b38e0055c146"). InnerVolumeSpecName "kube-api-access-x447p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:18.091754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.091727 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x447p\" (UniqueName: \"kubernetes.io/projected/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146-kube-api-access-x447p\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:18.714432 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.714399 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" Nov 02 21:45:18.714609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.714442 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm" event={"ID":"c65b3b06-2fdc-4cbe-b7e0-b38e0055c146","Type":"ContainerDied","Data":"2f5cc68688f0c8840d322888e683209c1bc78193bdc3903d31d635934b9633ac"} Nov 02 21:45:18.714609 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.714506 5859 scope.go:122] "RemoveContainer" containerID="56e74ad7960dba0e17278092adf7f63c5042787cd52434cf98e0d40de8e0c074" Nov 02 21:45:18.879873 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.879818 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:18.929643 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:18.929619 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-be6df0-29894265-chmcm"] Nov 02 21:45:20.579102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:20.579071 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" path="/var/lib/kubelet/pods/c65b3b06-2fdc-4cbe-b7e0-b38e0055c146/volumes" Nov 02 21:45:25.944542 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:25.944505 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:45:37.047383 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.047309 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:45:37.047866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.047527 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" podUID="f8c10886-3e07-418d-9954-c5a0c79cb359" containerName="maas-api" containerID="cri-o://e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13" gracePeriod=30 Nov 02 21:45:37.330929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.330905 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:45:37.409546 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.409517 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") pod \"f8c10886-3e07-418d-9954-c5a0c79cb359\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " Nov 02 21:45:37.409692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.409570 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\" (UniqueName: \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\") pod \"f8c10886-3e07-418d-9954-c5a0c79cb359\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " Nov 02 21:45:37.409771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.409757 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\") pod \"f8c10886-3e07-418d-9954-c5a0c79cb359\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " Nov 02 21:45:37.409887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.409874 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\") pod \"f8c10886-3e07-418d-9954-c5a0c79cb359\" (UID: \"f8c10886-3e07-418d-9954-c5a0c79cb359\") " Nov 02 21:45:37.410247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.410218 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca" pod "f8c10886-3e07-418d-9954-c5a0c79cb359" (UID: "f8c10886-3e07-418d-9954-c5a0c79cb359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:37.411633 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.411605 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs" pod "f8c10886-3e07-418d-9954-c5a0c79cb359" (UID: "f8c10886-3e07-418d-9954-c5a0c79cb359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:37.411731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.411677 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m" pod "f8c10886-3e07-418d-9954-c5a0c79cb359" (UID: "f8c10886-3e07-418d-9954-c5a0c79cb359"). InnerVolumeSpecName "kube-api-access-wnl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:37.411731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.411681 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls" pod "f8c10886-3e07-418d-9954-c5a0c79cb359" (UID: "f8c10886-3e07-418d-9954-c5a0c79cb359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:37.511011 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.510981 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8c10886-3e07-418d-9954-c5a0c79cb359-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:37.511011 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.511006 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:37.511011 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.511015 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl8m\" (UniqueName: \"kubernetes.io/projected/f8c10886-3e07-418d-9954-c5a0c79cb359-kube-api-access-wnl8m\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:37.511205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.511025 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8c10886-3e07-418d-9954-c5a0c79cb359-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:37.859101 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.859065 5859 generic.go:356] "Generic (PLEG): container finished" podID="f8c10886-3e07-418d-9954-c5a0c79cb359" containerID="e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13" exitCode=0 Nov 02 21:45:37.859260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.859110 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" Nov 02 21:45:37.859260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.859154 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" event={"ID":"f8c10886-3e07-418d-9954-c5a0c79cb359","Type":"ContainerDied","Data":"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13"} Nov 02 21:45:37.859260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.859201 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c" event={"ID":"f8c10886-3e07-418d-9954-c5a0c79cb359","Type":"ContainerDied","Data":"9af216d34f97bdbc914247935ffecfa232d8462a394922d8319fd98953cadfd9"} Nov 02 21:45:37.859260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.859226 5859 scope.go:122] "RemoveContainer" containerID="e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13" Nov 02 21:45:37.869244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.869225 5859 scope.go:122] "RemoveContainer" containerID="e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13" Nov 02 21:45:37.869504 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:37.869485 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13\": container with ID starting with e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13 not found: ID does not exist" containerID="e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13" Nov 02 21:45:37.869574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:37.869516 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13"} err="failed to get container status \"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13\": rpc error: code = NotFound desc = could not find container \"e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13\": container with ID starting with e4c2a43000f82b6c76289b7f5428c93cc06d2c8ecc9a61556092aad77543cb13 not found: ID does not exist" Nov 02 21:45:38.023902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:38.023882 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:45:38.026283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:38.026261 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-387f88-b54f9bf47-r9s4c"] Nov 02 21:45:38.571242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:38.571209 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c10886-3e07-418d-9954-c5a0c79cb359" path="/var/lib/kubelet/pods/f8c10886-3e07-418d-9954-c5a0c79cb359/volumes" Nov 02 21:45:49.357202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:49.357172 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:45:49.357658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:49.357389 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" containerID="cri-o://0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd" gracePeriod=30 Nov 02 21:45:51.316616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.316584 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:45:51.317207 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317192 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317208 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317220 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317224 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317257 5859 cpu_manager.go:414] "containerMap: removing container" podUID="da3d4288-e693-4feb-94be-52099a171618" containerName="maas-api" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317263 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3d4288-e693-4feb-94be-52099a171618" containerName="maas-api" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317279 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f8c10886-3e07-418d-9954-c5a0c79cb359" containerName="maas-api" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317284 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c10886-3e07-418d-9954-c5a0c79cb359" containerName="maas-api" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317365 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c10886-3e07-418d-9954-c5a0c79cb359" containerName="maas-api" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317377 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317383 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317390 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:45:51.317564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.317396 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3d4288-e693-4feb-94be-52099a171618" containerName="maas-api" Nov 02 21:45:51.320868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.320848 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.373963 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.373906 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.374265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.374227 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.374497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.374472 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.374691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.374665 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvx6\" (UniqueName: \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.416459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.416421 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:45:51.454807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.454780 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-aa60bc-serving-cert\"" Nov 02 21:45:51.476537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.476498 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.476835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.476803 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.477029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.477010 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.477247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.477171 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvx6\" (UniqueName: \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.478377 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:51.478347 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-serving-cert: secret "maas-api-e2e-ipp-b-aa60bc-serving-cert" not found Nov 02 21:45:51.478474 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:51.478437 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls podName:1b7bc367-c644-4422-a5bf-7b1f5227d73a nodeName:}" failed. No retries permitted until 2026-11-02 21:45:51.978415996 +0000 UTC m=+2191.526665772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls") pod "maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" (UID: "1b7bc367-c644-4422-a5bf-7b1f5227d73a") : secret "maas-api-e2e-ipp-b-aa60bc-serving-cert" not found Nov 02 21:45:51.479543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.479518 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.489163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.489137 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.535098 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.535060 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvx6\" (UniqueName: \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.985284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.985248 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:51.988225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:51.988196 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:52.235783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:52.235696 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:52.612861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:52.612834 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:45:52.614663 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:45:52.614628 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7bc367_c644_4422_a5bf_7b1f5227d73a.slice/crio-bf98b4946febc32d777cf4dc60ec541592d6aa5dac9d7a3de6f9007884ffc2f0 WatchSource:0}: Error finding container bf98b4946febc32d777cf4dc60ec541592d6aa5dac9d7a3de6f9007884ffc2f0: Status 404 returned error can't find the container with id bf98b4946febc32d777cf4dc60ec541592d6aa5dac9d7a3de6f9007884ffc2f0 Nov 02 21:45:52.971689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:52.971601 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" event={"ID":"1b7bc367-c644-4422-a5bf-7b1f5227d73a","Type":"ContainerStarted","Data":"bf98b4946febc32d777cf4dc60ec541592d6aa5dac9d7a3de6f9007884ffc2f0"} Nov 02 21:45:53.982878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:53.982836 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" event={"ID":"1b7bc367-c644-4422-a5bf-7b1f5227d73a","Type":"ContainerStarted","Data":"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d"} Nov 02 21:45:53.983331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:53.982920 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:45:54.101127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.101062 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" podStartSLOduration=2.173824469 podStartE2EDuration="3.101043496s" totalImagesPullingTime="927.219027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:54.097478981 +0000 UTC m=+2193.645728786" watchObservedRunningTime="2026-11-02 21:45:54.101043496 +0000 UTC m=+2193.649293278" Nov 02 21:45:54.633866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.633790 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:45:54.730544 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730495 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730562 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730600 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730643 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730762 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730851 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.730915 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730884 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.731110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.730913 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.731110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.731008 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\") pod \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\" (UID: \"5bebdf28-778c-4b94-babf-8162bf6ebe8c\") " Nov 02 21:45:54.732399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.732347 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:54.732399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.732372 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:54.732608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.732496 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:54.732608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.732565 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:54.732827 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.732800 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:54.735126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.735049 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:54.735256 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.735148 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:54.735311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.735253 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "kube-api-access-kk2qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:54.736484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.736462 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo" pod "5bebdf28-778c-4b94-babf-8162bf6ebe8c" (UID: "5bebdf28-778c-4b94-babf-8162bf6ebe8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:54.833152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833114 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833157 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2qx\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-kube-api-access-kk2qx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833172 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833186 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833201 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833214 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833227 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833242 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bebdf28-778c-4b94-babf-8162bf6ebe8c-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.833370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.833254 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bebdf28-778c-4b94-babf-8162bf6ebe8c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:54.990642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.990547 5859 generic.go:356] "Generic (PLEG): container finished" podID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerID="0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd" exitCode=0 Nov 02 21:45:54.990642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.990610 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" Nov 02 21:45:54.990642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.990629 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerDied","Data":"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd"} Nov 02 21:45:54.991236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.990680 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw" event={"ID":"5bebdf28-778c-4b94-babf-8162bf6ebe8c","Type":"ContainerDied","Data":"951979ea28f0571dfe94ce48bda796ee877f8ef01b6a73c7f24dce450966c201"} Nov 02 21:45:54.991236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:54.990691 5859 scope.go:122] "RemoveContainer" containerID="0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd" Nov 02 21:45:55.002857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.002839 5859 scope.go:122] "RemoveContainer" containerID="53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24" Nov 02 21:45:55.012682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.012655 5859 scope.go:122] "RemoveContainer" containerID="0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd" Nov 02 21:45:55.012960 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:55.012930 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd\": container with ID starting with 0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd not found: ID does not exist" containerID="0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd" Nov 02 21:45:55.013026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.012970 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd"} err="failed to get container status \"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd\": rpc error: code = NotFound desc = could not find container \"0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd\": container with ID starting with 0768c59ae2e310e797b10fef079d1a462e780a071a57e3ac6fdc12c3488670cd not found: ID does not exist" Nov 02 21:45:55.013026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.012992 5859 scope.go:122] "RemoveContainer" containerID="53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24" Nov 02 21:45:55.013238 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:55.013223 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24\": container with ID starting with 53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24 not found: ID does not exist" containerID="53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24" Nov 02 21:45:55.013285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.013243 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24"} err="failed to get container status \"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24\": rpc error: code = NotFound desc = could not find container \"53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24\": container with ID starting with 53bb9591c2b6e16d14aab64557a4214d94bed409ac0a5e33e068e4eefa108b24 not found: ID does not exist" Nov 02 21:45:55.158665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.158629 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:45:55.208488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:55.208455 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-387f88-openshift-default-86cd845f4f-cx2vw"] Nov 02 21:45:55.947341 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:45:55.947302 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:45:56.572139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:56.572092 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" path="/var/lib/kubelet/pods/5bebdf28-778c-4b94-babf-8162bf6ebe8c/volumes" Nov 02 21:45:59.996825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:45:59.996797 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:46:00.498986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.498939 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:46:00.499522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499510 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:00.499578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499524 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:00.499578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499564 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:46:00.499578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499570 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65b3b06-2fdc-4cbe-b7e0-b38e0055c146" containerName="cleanup" Nov 02 21:46:00.499671 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499653 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:00.499671 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.499668 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:00.504786 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.504769 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.601035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.601003 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-aa8aff\"" Nov 02 21:46:00.602087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.602025 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-aa8aff-dockercfg-pppjx\"" Nov 02 21:46:00.603945 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.603887 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:46:00.610845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.610815 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpqx\" (UniqueName: \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.611037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.610900 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.712427 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.712386 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpqx\" (UniqueName: \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.712617 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.712482 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.713187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.713168 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.770258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.770195 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpqx\" (UniqueName: \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\") pod \"payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:00.820076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:00.820052 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:01.191973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:01.191935 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:46:01.193101 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:46:01.193063 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf9ba23_e654_412e_ba41_e96a4e054da7.slice/crio-3975863ea770db9971a87aeadf8820f6060dad219e36bfcc80c1be9291b19bc8 WatchSource:0}: Error finding container 3975863ea770db9971a87aeadf8820f6060dad219e36bfcc80c1be9291b19bc8: Status 404 returned error can't find the container with id 3975863ea770db9971a87aeadf8820f6060dad219e36bfcc80c1be9291b19bc8 Nov 02 21:46:02.039016 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:02.038984 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" event={"ID":"3cf9ba23-e654-412e-ba41-e96a4e054da7","Type":"ContainerStarted","Data":"a46b2644b70b598c1e84aabad603d5b0859505ff2b23f05e4e3744948bceff56"} Nov 02 21:46:02.039197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:02.039045 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" event={"ID":"3cf9ba23-e654-412e-ba41-e96a4e054da7","Type":"ContainerStarted","Data":"3975863ea770db9971a87aeadf8820f6060dad219e36bfcc80c1be9291b19bc8"} Nov 02 21:46:02.039197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:02.039074 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:02.153327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:02.153276 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" podStartSLOduration=2.153260734 podStartE2EDuration="2.153260734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:02.15212105 +0000 UTC m=+2201.700370826" watchObservedRunningTime="2026-11-02 21:46:02.153260734 +0000 UTC m=+2201.701510513" Nov 02 21:46:05.118025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.117202 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:46:05.120100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.120072 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:05.120100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.120099 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bebdf28-778c-4b94-babf-8162bf6ebe8c" containerName="istio-proxy" Nov 02 21:46:05.125309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.125286 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.169050 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.169016 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:46:05.214825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.214796 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cc0d6a-dockercfg-lfzgc\"" Nov 02 21:46:05.214825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.214795 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cc0d6a\"" Nov 02 21:46:05.272614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.272585 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftsd\" (UniqueName: \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.272729 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.272640 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.374268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.374185 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftsd\" (UniqueName: \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.374268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.374254 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.374917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.374896 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.430926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.430892 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftsd\" (UniqueName: \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\") pod \"payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.454035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.454002 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:05.828665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:05.828615 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:46:05.830206 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:46:05.830178 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9338a6d1_f91b_41fb_a19e_d347233a8d1a.slice/crio-80d0229eff469eeec2c4e58bca015278e9f52d0c9386da49c41115e1e5b83022 WatchSource:0}: Error finding container 80d0229eff469eeec2c4e58bca015278e9f52d0c9386da49c41115e1e5b83022: Status 404 returned error can't find the container with id 80d0229eff469eeec2c4e58bca015278e9f52d0c9386da49c41115e1e5b83022 Nov 02 21:46:06.067808 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.067769 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" event={"ID":"9338a6d1-f91b-41fb-a19e-d347233a8d1a","Type":"ContainerStarted","Data":"edc736c0b2e166da291be07d7b20d7619cae1ef37073388f8fa338d9c1c04f46"} Nov 02 21:46:06.067994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.067837 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:06.067994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.067854 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" event={"ID":"9338a6d1-f91b-41fb-a19e-d347233a8d1a","Type":"ContainerStarted","Data":"80d0229eff469eeec2c4e58bca015278e9f52d0c9386da49c41115e1e5b83022"} Nov 02 21:46:06.183542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.183446 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" podStartSLOduration=1.183433114 podStartE2EDuration="1.183433114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:06.182834278 +0000 UTC m=+2205.731084058" watchObservedRunningTime="2026-11-02 21:46:06.183433114 +0000 UTC m=+2205.731682937" Nov 02 21:46:06.805448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.805418 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:46:06.805627 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.805597 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" containerID="cri-o://a6cd05ede80cc239c39eb4718845d4b0e8fe401b32d6954c62ec4970be0a348f" gracePeriod=30 Nov 02 21:46:06.854743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.854713 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:46:06.854990 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.854941 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" containerID="cri-o://0cb6bad1233308cc6dcc92f5995dc6ea26e789ab43138efa4ae34073dc366291" gracePeriod=30 Nov 02 21:46:06.857893 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.857867 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:46:06.858128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.858101 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" containerID="cri-o://cb51d478ac0105362ae226ab1cdc6414682cedf79a2fc06c6611d220394ea49f" gracePeriod=30 Nov 02 21:46:06.904916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.904891 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:46:06.905092 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:06.905074 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" containerID="cri-o://3489bbefb52c9563693ac873330b35d9ca2798bc9fb47bb12840ec33c9e498f1" gracePeriod=30 Nov 02 21:46:10.388463 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:10.388432 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-qkm7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:10.388913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:10.388482 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:12.112093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.111107 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" event={"ID":"98db4240-4786-41aa-8b51-eba1dec7b156","Type":"ContainerDied","Data":"0cb6bad1233308cc6dcc92f5995dc6ea26e789ab43138efa4ae34073dc366291"} Nov 02 21:46:12.112093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.111239 5859 generic.go:356] "Generic (PLEG): container finished" podID="98db4240-4786-41aa-8b51-eba1dec7b156" containerID="0cb6bad1233308cc6dcc92f5995dc6ea26e789ab43138efa4ae34073dc366291" exitCode=0 Nov 02 21:46:12.114979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.114501 5859 generic.go:356] "Generic (PLEG): container finished" podID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerID="a6cd05ede80cc239c39eb4718845d4b0e8fe401b32d6954c62ec4970be0a348f" exitCode=0 Nov 02 21:46:12.114979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.114611 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" event={"ID":"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec","Type":"ContainerDied","Data":"a6cd05ede80cc239c39eb4718845d4b0e8fe401b32d6954c62ec4970be0a348f"} Nov 02 21:46:12.117711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.117331 5859 generic.go:356] "Generic (PLEG): container finished" podID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerID="3489bbefb52c9563693ac873330b35d9ca2798bc9fb47bb12840ec33c9e498f1" exitCode=0 Nov 02 21:46:12.117895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.117871 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" event={"ID":"3504462b-dfd0-4cc7-ba4b-952574c2be7a","Type":"ContainerDied","Data":"3489bbefb52c9563693ac873330b35d9ca2798bc9fb47bb12840ec33c9e498f1"} Nov 02 21:46:12.121369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.121208 5859 generic.go:356] "Generic (PLEG): container finished" podID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerID="cb51d478ac0105362ae226ab1cdc6414682cedf79a2fc06c6611d220394ea49f" exitCode=0 Nov 02 21:46:12.121369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.121255 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" event={"ID":"b6590a39-bab1-46c2-9033-23d0dfa9302a","Type":"ContainerDied","Data":"cb51d478ac0105362ae226ab1cdc6414682cedf79a2fc06c6611d220394ea49f"} Nov 02 21:46:12.158438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.158407 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:46:12.191894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.191860 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:46:12.246065 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.246033 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:46:12.249435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.249411 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:46:12.251868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.251847 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.251994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.251977 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252016 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252049 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252082 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252162 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252232 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252284 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252259 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252295 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252317 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.252392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252346 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252375 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.252596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252429 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252474 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252507 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252538 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252602 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252607 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252652 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252670 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\") pod \"98db4240-4786-41aa-8b51-eba1dec7b156\" (UID: \"98db4240-4786-41aa-8b51-eba1dec7b156\") " Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252704 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.252799 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252763 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\") pod \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\" (UID: \"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec\") " Nov 02 21:46:12.253133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252802 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.253133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.252969 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.253614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253044 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.253614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253356 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:12.253614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253457 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.253614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253432 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.253614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253549 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.253895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253565 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.253895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253643 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.253895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253654 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.253895 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.253677 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.254600 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.254571 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "kube-api-access-gz6vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.254713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.254581 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.254826 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.254808 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.255440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.255407 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "kube-api-access-zfwlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.255537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.255511 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:12.255624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.255610 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:12.256482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.256462 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token" pod "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" (UID: "ce0d477e-a2b7-42d6-ba58-84053a0ac5ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.256666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.256646 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token" pod "98db4240-4786-41aa-8b51-eba1dec7b156" (UID: "98db4240-4786-41aa-8b51-eba1dec7b156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.354727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354628 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.354727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354681 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.354727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354715 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.354988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354781 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.354988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354813 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.354988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354901 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.354988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.354938 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355056 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.355204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355063 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.355204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355158 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355203 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355235 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355269 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355302 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355334 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.355369 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355340 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355435 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355446 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355521 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355549 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\") pod \"b6590a39-bab1-46c2-9033-23d0dfa9302a\" (UID: \"b6590a39-bab1-46c2-9033-23d0dfa9302a\") " Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355623 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355629 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.355666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355641 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.356061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355671 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\") pod \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\" (UID: \"3504462b-dfd0-4cc7-ba4b-952574c2be7a\") " Nov 02 21:46:12.356061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.355856 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.356191 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356159 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:12.356401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356374 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.356480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356443 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356463 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356479 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwlr\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-kube-api-access-zfwlr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356494 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356509 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356524 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98db4240-4786-41aa-8b51-eba1dec7b156-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356538 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356552 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98db4240-4786-41aa-8b51-eba1dec7b156-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356566 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356580 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.356595 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356595 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98db4240-4786-41aa-8b51-eba1dec7b156-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356612 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356625 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98db4240-4786-41aa-8b51-eba1dec7b156-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356638 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356653 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6590a39-bab1-46c2-9033-23d0dfa9302a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356667 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356680 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356694 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356707 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6vp\" (UniqueName: \"kubernetes.io/projected/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec-kube-api-access-gz6vp\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356722 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357128 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.356734 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.357922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.357785 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.357922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.357861 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.358771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.358714 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:12.358771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.358722 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.359194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.359167 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.359477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.359456 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:12.359775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.359742 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn" pod "3504462b-dfd0-4cc7-ba4b-952574c2be7a" (UID: "3504462b-dfd0-4cc7-ba4b-952574c2be7a"). InnerVolumeSpecName "kube-api-access-chshn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.359878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.359822 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:12.360002 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.359985 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw" pod "b6590a39-bab1-46c2-9033-23d0dfa9302a" (UID: "b6590a39-bab1-46c2-9033-23d0dfa9302a"). InnerVolumeSpecName "kube-api-access-ncwxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:12.458058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458022 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458051 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3504462b-dfd0-4cc7-ba4b-952574c2be7a-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458061 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458071 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458079 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3504462b-dfd0-4cc7-ba4b-952574c2be7a-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458088 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chshn\" (UniqueName: \"kubernetes.io/projected/3504462b-dfd0-4cc7-ba4b-952574c2be7a-kube-api-access-chshn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458097 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458105 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwxw\" (UniqueName: \"kubernetes.io/projected/b6590a39-bab1-46c2-9033-23d0dfa9302a-kube-api-access-ncwxw\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458113 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:12.458304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:12.458122 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6590a39-bab1-46c2-9033-23d0dfa9302a-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:13.049260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.049229 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:46:13.129202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.129105 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" event={"ID":"b6590a39-bab1-46c2-9033-23d0dfa9302a","Type":"ContainerDied","Data":"d0097fdddb2529f2b2f15be4dae72cc6cbd0d8f19688a8de9f71f6586c01569e"} Nov 02 21:46:13.129202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.129176 5859 scope.go:122] "RemoveContainer" containerID="cb51d478ac0105362ae226ab1cdc6414682cedf79a2fc06c6611d220394ea49f" Nov 02 21:46:13.129202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.129111 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc" Nov 02 21:46:13.130762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.130693 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" event={"ID":"98db4240-4786-41aa-8b51-eba1dec7b156","Type":"ContainerDied","Data":"e226c764ce40a2c05441529e75a0586bc46b731ca3a21d2f977d080bfc46fd9f"} Nov 02 21:46:13.130829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.130771 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n" Nov 02 21:46:13.132598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.132580 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" Nov 02 21:46:13.133102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.133082 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9" event={"ID":"ce0d477e-a2b7-42d6-ba58-84053a0ac5ec","Type":"ContainerDied","Data":"19c6586c1d91514097f0eeb23d2df75f6af4b3057a850c5ab7348aef1c042655"} Nov 02 21:46:13.134581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.134542 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" Nov 02 21:46:13.134581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.134557 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz" event={"ID":"3504462b-dfd0-4cc7-ba4b-952574c2be7a","Type":"ContainerDied","Data":"2eda10484f77d3d32785ca02d02813a7322168d925fe708c5065df52bf1b5407"} Nov 02 21:46:13.143890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.143825 5859 scope.go:122] "RemoveContainer" containerID="0cb6bad1233308cc6dcc92f5995dc6ea26e789ab43138efa4ae34073dc366291" Nov 02 21:46:13.159543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.159513 5859 scope.go:122] "RemoveContainer" containerID="a6cd05ede80cc239c39eb4718845d4b0e8fe401b32d6954c62ec4970be0a348f" Nov 02 21:46:13.171569 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.171543 5859 scope.go:122] "RemoveContainer" containerID="3489bbefb52c9563693ac873330b35d9ca2798bc9fb47bb12840ec33c9e498f1" Nov 02 21:46:13.278171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.278137 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:46:13.280716 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.280695 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ckwj9"] Nov 02 21:46:13.391203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.391113 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:46:13.398850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.398813 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-qkm7n"] Nov 02 21:46:13.507688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.507653 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:46:13.510222 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.510200 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-w8bqz"] Nov 02 21:46:13.618744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.618712 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:46:13.621310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:13.621288 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-v7zfc"] Nov 02 21:46:14.571301 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:14.571263 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" path="/var/lib/kubelet/pods/3504462b-dfd0-4cc7-ba4b-952574c2be7a/volumes" Nov 02 21:46:14.571758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:14.571743 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" path="/var/lib/kubelet/pods/98db4240-4786-41aa-8b51-eba1dec7b156/volumes" Nov 02 21:46:14.572196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:14.572181 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" path="/var/lib/kubelet/pods/b6590a39-bab1-46c2-9033-23d0dfa9302a/volumes" Nov 02 21:46:14.572927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:14.572910 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" path="/var/lib/kubelet/pods/ce0d477e-a2b7-42d6-ba58-84053a0ac5ec/volumes" Nov 02 21:46:17.076697 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:17.076658 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:46:25.944289 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:46:25.944255 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:46:27.959592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.959558 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:46:27.960121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960099 5859 cpu_manager.go:414] "containerMap: removing container" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960121 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960138 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960143 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960154 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960158 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" Nov 02 21:46:27.960179 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960178 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" Nov 02 21:46:27.960359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960182 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" Nov 02 21:46:27.960359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960277 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="98db4240-4786-41aa-8b51-eba1dec7b156" containerName="istio-proxy" Nov 02 21:46:27.960359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960285 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3504462b-dfd0-4cc7-ba4b-952574c2be7a" containerName="istio-proxy" Nov 02 21:46:27.960359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960293 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6590a39-bab1-46c2-9033-23d0dfa9302a" containerName="istio-proxy" Nov 02 21:46:27.960359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.960303 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0d477e-a2b7-42d6-ba58-84053a0ac5ec" containerName="istio-proxy" Nov 02 21:46:27.965110 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:27.965093 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.017301 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.017275 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:46:28.064480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.064450 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8e90da-serving-cert\"" Nov 02 21:46:28.140707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.140680 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.140844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.140717 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.140844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.140740 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.140919 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.140838 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dgs\" (UniqueName: \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.242394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.242316 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.242394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.242358 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.242394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.242380 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.242656 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:46:28.242467 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-serving-cert: secret "maas-api-e2e-praxis-a-8e90da-serving-cert" not found Nov 02 21:46:28.242656 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.242510 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dgs\" (UniqueName: \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.242656 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:46:28.242523 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls podName:b1a719b0-bead-4e9b-9b2e-e5a5f7121048 nodeName:}" failed. No retries permitted until 2026-11-02 21:46:28.742508444 +0000 UTC m=+2228.290758217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls") pod "maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" (UID: "b1a719b0-bead-4e9b-9b2e-e5a5f7121048") : secret "maas-api-e2e-praxis-a-8e90da-serving-cert" not found Nov 02 21:46:28.243007 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.242989 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.244962 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.244929 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.303037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.303011 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dgs\" (UniqueName: \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.747947 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.747912 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.750259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.750237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:28.878074 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:28.878051 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:29.256928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:29.256899 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:46:29.258172 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:46:29.258136 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a719b0_bead_4e9b_9b2e_e5a5f7121048.slice/crio-59cefe8d5f4ef7c7eb93a4a451b9a15bd53996e6d3d0c808caba6407c0c61b68 WatchSource:0}: Error finding container 59cefe8d5f4ef7c7eb93a4a451b9a15bd53996e6d3d0c808caba6407c0c61b68: Status 404 returned error can't find the container with id 59cefe8d5f4ef7c7eb93a4a451b9a15bd53996e6d3d0c808caba6407c0c61b68 Nov 02 21:46:30.253573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:30.253540 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" event={"ID":"b1a719b0-bead-4e9b-9b2e-e5a5f7121048","Type":"ContainerStarted","Data":"59cefe8d5f4ef7c7eb93a4a451b9a15bd53996e6d3d0c808caba6407c0c61b68"} Nov 02 21:46:31.261194 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:31.261156 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" event={"ID":"b1a719b0-bead-4e9b-9b2e-e5a5f7121048","Type":"ContainerStarted","Data":"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f"} Nov 02 21:46:31.261647 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:31.261228 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:31.376663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:31.376615 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" podStartSLOduration=3.284431464 podStartE2EDuration="4.376602234s" totalImagesPullingTime="1.09217077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:31.374702751 +0000 UTC m=+2230.922952531" watchObservedRunningTime="2026-11-02 21:46:31.376602234 +0000 UTC m=+2230.924852014" Nov 02 21:46:37.282457 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:46:37.282421 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:46:55.944522 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:46:55.944487 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:47:03.893416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:03.893339 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:47:03.897739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:03.897720 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:03.953648 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:03.953618 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:47:04.000832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.000804 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-aa8aff-serving-cert\"" Nov 02 21:47:04.001007 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.000983 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.001068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.001028 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.001127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.001088 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg45h\" (UniqueName: \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.001236 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.001213 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.101869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.101841 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.102038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.101876 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.102038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.101911 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg45h\" (UniqueName: \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.102038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.101975 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.102224 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:04.102107 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-serving-cert: secret "maas-api-e2e-praxis-a-aa8aff-serving-cert" not found Nov 02 21:47:04.102224 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:04.102172 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls podName:8b03d709-7126-4207-85a4-92a861f77f3c nodeName:}" failed. No retries permitted until 2026-11-02 21:47:04.602156103 +0000 UTC m=+2264.150405879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls") pod "maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" (UID: "8b03d709-7126-4207-85a4-92a861f77f3c") : secret "maas-api-e2e-praxis-a-aa8aff-serving-cert" not found Nov 02 21:47:04.102574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.102550 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.104292 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.104271 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.161907 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.161837 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg45h\" (UniqueName: \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.607834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.607805 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.610199 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.610175 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:04.810845 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:04.810820 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:05.241651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:05.241627 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:47:05.243907 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:47:05.243880 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b03d709_7126_4207_85a4_92a861f77f3c.slice/crio-efa7342d34deaa674a562c56e225a0d0c9e74c125d0b56a4397de562a743c729 WatchSource:0}: Error finding container efa7342d34deaa674a562c56e225a0d0c9e74c125d0b56a4397de562a743c729: Status 404 returned error can't find the container with id efa7342d34deaa674a562c56e225a0d0c9e74c125d0b56a4397de562a743c729 Nov 02 21:47:05.245627 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:05.245609 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:47:05.469966 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:05.469924 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" event={"ID":"8b03d709-7126-4207-85a4-92a861f77f3c","Type":"ContainerStarted","Data":"efa7342d34deaa674a562c56e225a0d0c9e74c125d0b56a4397de562a743c729"} Nov 02 21:47:06.478098 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:06.478057 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" event={"ID":"8b03d709-7126-4207-85a4-92a861f77f3c","Type":"ContainerStarted","Data":"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430"} Nov 02 21:47:06.478478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:06.478227 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:06.612243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:06.612162 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" podStartSLOduration=2.526031367 podStartE2EDuration="3.612148496s" totalImagesPullingTime="1.086117129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:06.610811199 +0000 UTC m=+2266.159060981" watchObservedRunningTime="2026-11-02 21:47:06.612148496 +0000 UTC m=+2266.160398277" Nov 02 21:47:07.980027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:07.979996 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:47:12.490327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:12.490294 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:47:12.980640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:12.980596 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" podUID="a2653f2d-c710-4cbf-b119-1a1237e3c523" containerName="payload-pre-processing" containerID="cri-o://a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc" gracePeriod=25 Nov 02 21:47:13.256004 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.255979 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:47:13.302677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.302654 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\" (UniqueName: \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\") pod \"a2653f2d-c710-4cbf-b119-1a1237e3c523\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " Nov 02 21:47:13.302787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.302693 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\") pod \"a2653f2d-c710-4cbf-b119-1a1237e3c523\" (UID: \"a2653f2d-c710-4cbf-b119-1a1237e3c523\") " Nov 02 21:47:13.303117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.303086 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume" pod "a2653f2d-c710-4cbf-b119-1a1237e3c523" (UID: "a2653f2d-c710-4cbf-b119-1a1237e3c523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:13.304674 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.304652 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h" pod "a2653f2d-c710-4cbf-b119-1a1237e3c523" (UID: "a2653f2d-c710-4cbf-b119-1a1237e3c523"). InnerVolumeSpecName "kube-api-access-xf22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:13.403933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.403906 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf22h\" (UniqueName: \"kubernetes.io/projected/a2653f2d-c710-4cbf-b119-1a1237e3c523-kube-api-access-xf22h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:13.403933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.403929 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2653f2d-c710-4cbf-b119-1a1237e3c523-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:13.519132 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.519062 5859 generic.go:356] "Generic (PLEG): container finished" podID="a2653f2d-c710-4cbf-b119-1a1237e3c523" containerID="a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc" exitCode=0 Nov 02 21:47:13.519545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.519131 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" Nov 02 21:47:13.519545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.519149 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" event={"ID":"a2653f2d-c710-4cbf-b119-1a1237e3c523","Type":"ContainerDied","Data":"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc"} Nov 02 21:47:13.519545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.519215 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn" event={"ID":"a2653f2d-c710-4cbf-b119-1a1237e3c523","Type":"ContainerDied","Data":"4195eaadb806fc4dc1c98f206ba566c54a1dabf55957c565490685e454836957"} Nov 02 21:47:13.519545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.519248 5859 scope.go:122] "RemoveContainer" containerID="a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc" Nov 02 21:47:13.530303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.530287 5859 scope.go:122] "RemoveContainer" containerID="a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc" Nov 02 21:47:13.530546 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:13.530532 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc\": container with ID starting with a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc not found: ID does not exist" containerID="a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc" Nov 02 21:47:13.530586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.530554 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc"} err="failed to get container status \"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc\": rpc error: code = NotFound desc = could not find container \"a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc\": container with ID starting with a2b05bcb5f9db613d86d379042111c84a4e2061aa8a40bbd21301aa4253dcffc not found: ID does not exist" Nov 02 21:47:13.683642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.683610 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:47:13.689363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:13.689335 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa60bc-9d448554c-8sprn"] Nov 02 21:47:14.570530 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:14.570502 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2653f2d-c710-4cbf-b119-1a1237e3c523" path="/var/lib/kubelet/pods/a2653f2d-c710-4cbf-b119-1a1237e3c523/volumes" Nov 02 21:47:25.943863 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:25.943829 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:47:50.032465 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.032428 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:47:50.035938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.033191 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" podUID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" containerName="maas-api" containerID="cri-o://c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd" gracePeriod=30 Nov 02 21:47:50.316712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.316681 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:47:50.392731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.392705 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\" (UniqueName: \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\") pod \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " Nov 02 21:47:50.392858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.392747 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\") pod \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " Nov 02 21:47:50.392858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.392781 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\") pod \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " Nov 02 21:47:50.392991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.392885 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") pod \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\" (UID: \"7fb46b67-2a9c-4903-a3ff-5384c9fb8187\") " Nov 02 21:47:50.393218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.393191 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca" pod "7fb46b67-2a9c-4903-a3ff-5384c9fb8187" (UID: "7fb46b67-2a9c-4903-a3ff-5384c9fb8187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:50.393379 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.393349 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:50.394913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.394870 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg" pod "7fb46b67-2a9c-4903-a3ff-5384c9fb8187" (UID: "7fb46b67-2a9c-4903-a3ff-5384c9fb8187"). InnerVolumeSpecName "kube-api-access-2t9pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:50.395046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.395026 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs" pod "7fb46b67-2a9c-4903-a3ff-5384c9fb8187" (UID: "7fb46b67-2a9c-4903-a3ff-5384c9fb8187"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:50.395195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.395168 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls" pod "7fb46b67-2a9c-4903-a3ff-5384c9fb8187" (UID: "7fb46b67-2a9c-4903-a3ff-5384c9fb8187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:50.494608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.494558 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:50.494608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.494607 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9pg\" (UniqueName: \"kubernetes.io/projected/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-kube-api-access-2t9pg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:50.494815 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.494619 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb46b67-2a9c-4903-a3ff-5384c9fb8187-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:50.753506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.753473 5859 generic.go:356] "Generic (PLEG): container finished" podID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" containerID="c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd" exitCode=0 Nov 02 21:47:50.753686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.753515 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" Nov 02 21:47:50.753686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.753557 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" event={"ID":"7fb46b67-2a9c-4903-a3ff-5384c9fb8187","Type":"ContainerDied","Data":"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd"} Nov 02 21:47:50.753686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.753612 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr" event={"ID":"7fb46b67-2a9c-4903-a3ff-5384c9fb8187","Type":"ContainerDied","Data":"463a42d4e03b92d59091b97c1500b67a0e45f452ba8e87e1917c95a3f1f41aad"} Nov 02 21:47:50.753686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.753640 5859 scope.go:122] "RemoveContainer" containerID="c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd" Nov 02 21:47:50.766459 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.766443 5859 scope.go:122] "RemoveContainer" containerID="c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd" Nov 02 21:47:50.766735 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:50.766715 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd\": container with ID starting with c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd not found: ID does not exist" containerID="c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd" Nov 02 21:47:50.766800 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.766742 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd"} err="failed to get container status \"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd\": rpc error: code = NotFound desc = could not find container \"c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd\": container with ID starting with c4f80c5234eb7fd9e1ce3b78fda0e1ed36f97661807691b3d9ee611d0aca6edd not found: ID does not exist" Nov 02 21:47:50.917519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.917463 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:47:50.920777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:50.920746 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e12f99-9d898488b-58rhr"] Nov 02 21:47:51.395773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.395739 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:47:51.396309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.396009 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" podUID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" containerName="maas-api" containerID="cri-o://30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6" gracePeriod=30 Nov 02 21:47:51.671254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.671230 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:47:51.761253 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.761220 5859 generic.go:356] "Generic (PLEG): container finished" podID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" containerID="30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6" exitCode=0 Nov 02 21:47:51.761392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.761261 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" Nov 02 21:47:51.761392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.761303 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" event={"ID":"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c","Type":"ContainerDied","Data":"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6"} Nov 02 21:47:51.761392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.761356 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s" event={"ID":"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c","Type":"ContainerDied","Data":"7f51f8e166738023936e40d3550a65714f29299f57e0d449f72556a79c1d3a46"} Nov 02 21:47:51.761392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.761383 5859 scope.go:122] "RemoveContainer" containerID="30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6" Nov 02 21:47:51.771278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.771260 5859 scope.go:122] "RemoveContainer" containerID="30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6" Nov 02 21:47:51.771584 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:51.771559 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6\": container with ID starting with 30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6 not found: ID does not exist" containerID="30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6" Nov 02 21:47:51.771665 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.771592 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6"} err="failed to get container status \"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6\": rpc error: code = NotFound desc = could not find container \"30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6\": container with ID starting with 30e6c5f3f898465391b7a5f0e70e79c994757eeb828ae8faddb30ff1511cf0d6 not found: ID does not exist" Nov 02 21:47:51.816322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816289 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\") pod \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " Nov 02 21:47:51.816505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816355 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\" (UniqueName: \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\") pod \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " Nov 02 21:47:51.816505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816449 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\") pod \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " Nov 02 21:47:51.816629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816524 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") pod \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\" (UID: \"d69aec57-2fc4-4ca6-8c89-2e1923d64e4c\") " Nov 02 21:47:51.816884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816852 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca" pod "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" (UID: "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:51.817044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.816991 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:51.818660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.818626 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls" pod "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" (UID: "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:51.818660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.818653 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs" pod "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" (UID: "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:51.818823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.818692 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h" pod "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" (UID: "d69aec57-2fc4-4ca6-8c89-2e1923d64e4c"). InnerVolumeSpecName "kube-api-access-vzk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:51.918210 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.918125 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:51.918210 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.918160 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzk8h\" (UniqueName: \"kubernetes.io/projected/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-kube-api-access-vzk8h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:51.918210 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:51.918170 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:52.183943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:52.183868 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:47:52.186217 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:52.186198 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ce2453-56d6956ccb-s6h6s"] Nov 02 21:47:52.572582 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:52.572548 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" path="/var/lib/kubelet/pods/7fb46b67-2a9c-4903-a3ff-5384c9fb8187/volumes" Nov 02 21:47:52.573035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:52.573021 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" path="/var/lib/kubelet/pods/d69aec57-2fc4-4ca6-8c89-2e1923d64e4c/volumes" Nov 02 21:47:54.947686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.947659 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:47:54.948262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948246 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" containerName="maas-api" Nov 02 21:47:54.948262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948263 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" containerName="maas-api" Nov 02 21:47:54.948352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948282 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a2653f2d-c710-4cbf-b119-1a1237e3c523" containerName="payload-pre-processing" Nov 02 21:47:54.948352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948287 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2653f2d-c710-4cbf-b119-1a1237e3c523" containerName="payload-pre-processing" Nov 02 21:47:54.948352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948325 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" containerName="maas-api" Nov 02 21:47:54.948352 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948331 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" containerName="maas-api" Nov 02 21:47:54.948496 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948430 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb46b67-2a9c-4903-a3ff-5384c9fb8187" containerName="maas-api" Nov 02 21:47:54.948496 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948444 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2653f2d-c710-4cbf-b119-1a1237e3c523" containerName="payload-pre-processing" Nov 02 21:47:54.948496 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.948456 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69aec57-2fc4-4ca6-8c89-2e1923d64e4c" containerName="maas-api" Nov 02 21:47:54.953120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:54.953103 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.006367 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.006316 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-942164-openshift-default-dockercfg-4xmth\"" Nov 02 21:47:55.051741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.051707 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.051741 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.051746 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.051773 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052001 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052147 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052180 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052216 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052316 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052294 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblmh\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.052421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.052407 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.059529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.059503 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:47:55.153903 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.153866 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblmh\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154008 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154064 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154089 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154127 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154117 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154188 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154291 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154318 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154354 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154504 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154639 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154607 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154702 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154663 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154730 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.154847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.154828 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.156453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.156432 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.156803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.156784 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.210985 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.210906 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblmh\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.211131 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.211112 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\") pod \"e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.276160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.276136 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:55.702635 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.702606 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:47:55.704395 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:47:55.704372 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ee309f_b111_403b_9e19_846981c524d3.slice/crio-a58ffa8eb24507d6a3fe18630355830df16b628687ac37776bf1ff3da317e962 WatchSource:0}: Error finding container a58ffa8eb24507d6a3fe18630355830df16b628687ac37776bf1ff3da317e962: Status 404 returned error can't find the container with id a58ffa8eb24507d6a3fe18630355830df16b628687ac37776bf1ff3da317e962 Nov 02 21:47:55.706670 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.706629 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:47:55.706762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.706704 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:47:55.706762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.706743 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:47:55.793645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.793608 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerStarted","Data":"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e"} Nov 02 21:47:55.793820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.793665 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerStarted","Data":"a58ffa8eb24507d6a3fe18630355830df16b628687ac37776bf1ff3da317e962"} Nov 02 21:47:55.912832 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:55.912785 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podStartSLOduration=1.912771529 podStartE2EDuration="1.912771529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:55.910164995 +0000 UTC m=+2315.458414776" watchObservedRunningTime="2026-11-02 21:47:55.912771529 +0000 UTC m=+2315.461021309" Nov 02 21:47:55.947652 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:47:55.947618 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:47:56.276587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.276555 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:56.282310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.282287 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:56.801312 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.801282 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:56.802299 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.802275 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:47:56.802894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.802860 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:47:56.803076 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:47:56.803051 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" containerID="cri-o://cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79" gracePeriod=30 Nov 02 21:48:02.069616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.069557 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:48:02.240538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240508 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.240773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240550 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.240773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240580 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.240773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240695 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.240773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240743 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.240773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.240769 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.241410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241311 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.241410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241327 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:02.241410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241406 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.241781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241450 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\") pod \"cc97b16c-7401-43eb-96a3-db052723e6cb\" (UID: \"cc97b16c-7401-43eb-96a3-db052723e6cb\") " Nov 02 21:48:02.241781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241549 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:02.241901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241831 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:02.241901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241870 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:02.241901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.241784 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:02.242129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.242109 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc97b16c-7401-43eb-96a3-db052723e6cb-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.242129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.242128 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.242309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.242137 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.242309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.242146 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.242309 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.242154 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.243085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.243065 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:02.243743 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.243717 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "kube-api-access-9b5xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:02.244136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.244108 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:02.244214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.244198 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token" pod "cc97b16c-7401-43eb-96a3-db052723e6cb" (UID: "cc97b16c-7401-43eb-96a3-db052723e6cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:02.343566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.343518 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5xp\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-kube-api-access-9b5xp\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.343566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.343543 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.343566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.343554 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.343566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.343563 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc97b16c-7401-43eb-96a3-db052723e6cb-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:02.845384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.845337 5859 generic.go:356] "Generic (PLEG): container finished" podID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerID="cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79" exitCode=0 Nov 02 21:48:02.845384 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.845384 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerDied","Data":"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79"} Nov 02 21:48:02.845640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.845408 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" Nov 02 21:48:02.845640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.845432 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq" event={"ID":"cc97b16c-7401-43eb-96a3-db052723e6cb","Type":"ContainerDied","Data":"9363e2411ba89093f43e2ca3e3a1191a7c29a1fe61e13c16af113b92a35e991e"} Nov 02 21:48:02.845640 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.845458 5859 scope.go:122] "RemoveContainer" containerID="cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79" Nov 02 21:48:02.860910 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.860870 5859 scope.go:122] "RemoveContainer" containerID="363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd" Nov 02 21:48:02.871340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.871321 5859 scope.go:122] "RemoveContainer" containerID="cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79" Nov 02 21:48:02.871619 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:48:02.871600 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79\": container with ID starting with cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79 not found: ID does not exist" containerID="cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79" Nov 02 21:48:02.871672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.871636 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79"} err="failed to get container status \"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79\": rpc error: code = NotFound desc = could not find container \"cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79\": container with ID starting with cb47d6c907ea52cf456dbe21dda1140ada072d8a0cbc6d42a44a0476204b7d79 not found: ID does not exist" Nov 02 21:48:02.871672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.871654 5859 scope.go:122] "RemoveContainer" containerID="363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd" Nov 02 21:48:02.871966 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:48:02.871936 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd\": container with ID starting with 363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd not found: ID does not exist" containerID="363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd" Nov 02 21:48:02.872063 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.871991 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd"} err="failed to get container status \"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd\": rpc error: code = NotFound desc = could not find container \"363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd\": container with ID starting with 363a3afcc56962c19f288ac65595d8f181c9e5d316ca527d85229c8105253bfd not found: ID does not exist" Nov 02 21:48:02.971223 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.971195 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:48:02.971917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.971893 5859 cpu_manager.go:414] "containerMap: removing container" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:02.971917 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.971916 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:02.972171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.972155 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:02.972240 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.972174 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:02.977255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:02.977238 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.011792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.011769 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:48:03.021376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.021358 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ce2453-openshift-default-57bdb46c55-8fhwq"] Nov 02 21:48:03.030044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.030022 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:48:03.062991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.059428 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-942164\"" Nov 02 21:48:03.062991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.059604 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-942164-dockercfg-bsv6w\"" Nov 02 21:48:03.154265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.154175 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w988\" (UniqueName: \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.154265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.154212 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.255437 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.255409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w988\" (UniqueName: \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.255599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.255445 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.256172 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.256154 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.312337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.312308 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w988\" (UniqueName: \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\") pod \"payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.589834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.589806 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:03.967282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:03.967256 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:48:03.969457 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:48:03.969429 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad17068f_1f5a_4004_8bdf_303d57860eb4.slice/crio-7f72c270d8934dd6d8f5331c0a54fc764d253c8b53c63cda5b618f1392236313 WatchSource:0}: Error finding container 7f72c270d8934dd6d8f5331c0a54fc764d253c8b53c63cda5b618f1392236313: Status 404 returned error can't find the container with id 7f72c270d8934dd6d8f5331c0a54fc764d253c8b53c63cda5b618f1392236313 Nov 02 21:48:04.571122 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:04.571090 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" path="/var/lib/kubelet/pods/cc97b16c-7401-43eb-96a3-db052723e6cb/volumes" Nov 02 21:48:04.859885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:04.859816 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" event={"ID":"ad17068f-1f5a-4004-8bdf-303d57860eb4","Type":"ContainerStarted","Data":"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6"} Nov 02 21:48:04.859885 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:04.859865 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" event={"ID":"ad17068f-1f5a-4004-8bdf-303d57860eb4","Type":"ContainerStarted","Data":"7f72c270d8934dd6d8f5331c0a54fc764d253c8b53c63cda5b618f1392236313"} Nov 02 21:48:04.860095 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:04.859894 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:04.975700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:04.975645 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" podStartSLOduration=2.975631769 podStartE2EDuration="2.975631769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:04.973473337 +0000 UTC m=+2324.521723118" watchObservedRunningTime="2026-11-02 21:48:04.975631769 +0000 UTC m=+2324.523881549" Nov 02 21:48:15.869185 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:15.869159 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:48:25.758037 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.758004 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:48:25.758562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.758549 5859 cpu_manager.go:414] "containerMap: removing container" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:25.758607 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.758564 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc97b16c-7401-43eb-96a3-db052723e6cb" containerName="istio-proxy" Nov 02 21:48:25.763419 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.763402 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.817328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.817303 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bb45bc-openshift-default-dockercfg-vvrc4\"" Nov 02 21:48:25.869513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.869485 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:48:25.874232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874207 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgc6b\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874244 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874269 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874325 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874489 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874594 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874631 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874751 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.874824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.874802 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.945897 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:48:25.945863 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:48:25.975866 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.975836 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.975883 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.975972 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976023 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976083 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgc6b\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976126 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976158 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976255 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976182 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976266 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976496 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976634 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976752 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976707 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.976896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.976878 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.977025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.977002 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.978505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.978483 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:25.978806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:25.978786 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:26.033679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.033619 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:26.033782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.033695 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgc6b\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\") pod \"e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:26.077216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.077183 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:26.505748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.505720 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:48:26.507114 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:48:26.507081 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c57a72_83b4_4ac8_9ed7_17ff0eba540f.slice/crio-ccd8df8a8bbb507c56d0f365921de8c6540008c6a1423f83a1abf39a744975ea WatchSource:0}: Error finding container ccd8df8a8bbb507c56d0f365921de8c6540008c6a1423f83a1abf39a744975ea: Status 404 returned error can't find the container with id ccd8df8a8bbb507c56d0f365921de8c6540008c6a1423f83a1abf39a744975ea Nov 02 21:48:26.509472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.509439 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:26.509567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.509503 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:26.509567 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.509532 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:48:26.999469 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.999430 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerStarted","Data":"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec"} Nov 02 21:48:26.999848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:26.999487 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerStarted","Data":"ccd8df8a8bbb507c56d0f365921de8c6540008c6a1423f83a1abf39a744975ea"} Nov 02 21:48:27.077989 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:27.077965 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:27.082807 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:27.082787 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:27.125855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:27.125814 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podStartSLOduration=2.125802271 podStartE2EDuration="2.125802271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:27.121693288 +0000 UTC m=+2346.669943069" watchObservedRunningTime="2026-11-02 21:48:27.125802271 +0000 UTC m=+2346.674052051" Nov 02 21:48:28.005228 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:28.005197 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:28.005932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:28.005914 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:48:33.460652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.460559 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:48:33.466698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.466664 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.518418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.518359 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:48:33.561020 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.560980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9l9b\" (UniqueName: \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.561268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.561244 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.564453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.564418 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bb45bc\"" Nov 02 21:48:33.564579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.564531 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bb45bc-dockercfg-69hl7\"" Nov 02 21:48:33.620884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.620856 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:48:33.625129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.625113 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.662586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.662557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.662739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.662691 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9l9b\" (UniqueName: \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.663258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.663239 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.677925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.677900 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:48:33.719835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.719786 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9l9b\" (UniqueName: \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\") pod \"payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.763500 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.763474 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.763584 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.763534 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfcns\" (UniqueName: \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.782993 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.782946 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:33.864757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.864730 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.864880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.864826 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfcns\" (UniqueName: \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.865616 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.865590 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.921889 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.921863 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfcns\" (UniqueName: \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\") pod \"payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:33.938932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:33.938911 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:34.154588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:34.154563 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:48:34.155477 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:48:34.155453 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c436fcd_a610_4835_b495_016edb69349c.slice/crio-1ce0944e8fbf2bc129b6ae32807ec4b234cffeb4484793af8dc070240443cff5 WatchSource:0}: Error finding container 1ce0944e8fbf2bc129b6ae32807ec4b234cffeb4484793af8dc070240443cff5: Status 404 returned error can't find the container with id 1ce0944e8fbf2bc129b6ae32807ec4b234cffeb4484793af8dc070240443cff5 Nov 02 21:48:34.310746 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:48:34.310714 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee42edad_5982_436e_a25f_edd440443331.slice/crio-29cdaa4819005d4fc3fd24612fb6ca50ef9b34a7d991a18e5d5b42f89fdbae26 WatchSource:0}: Error finding container 29cdaa4819005d4fc3fd24612fb6ca50ef9b34a7d991a18e5d5b42f89fdbae26: Status 404 returned error can't find the container with id 29cdaa4819005d4fc3fd24612fb6ca50ef9b34a7d991a18e5d5b42f89fdbae26 Nov 02 21:48:34.311365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:34.311282 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:48:35.049718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.049687 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" event={"ID":"ee42edad-5982-436e-a25f-edd440443331","Type":"ContainerStarted","Data":"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94"} Nov 02 21:48:35.050157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.049742 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" event={"ID":"ee42edad-5982-436e-a25f-edd440443331","Type":"ContainerStarted","Data":"29cdaa4819005d4fc3fd24612fb6ca50ef9b34a7d991a18e5d5b42f89fdbae26"} Nov 02 21:48:35.050157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.049783 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:35.051396 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.051374 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" event={"ID":"4c436fcd-a610-4835-b495-016edb69349c","Type":"ContainerStarted","Data":"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc"} Nov 02 21:48:35.051793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.051412 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" event={"ID":"4c436fcd-a610-4835-b495-016edb69349c","Type":"ContainerStarted","Data":"1ce0944e8fbf2bc129b6ae32807ec4b234cffeb4484793af8dc070240443cff5"} Nov 02 21:48:35.051793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.051453 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:35.165841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.165796 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" podStartSLOduration=2.16578567 podStartE2EDuration="2.16578567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:35.164780574 +0000 UTC m=+2354.713030379" watchObservedRunningTime="2026-11-02 21:48:35.16578567 +0000 UTC m=+2354.714035450" Nov 02 21:48:35.283658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:35.283526 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" podStartSLOduration=2.283508589 podStartE2EDuration="2.283508589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:35.282551639 +0000 UTC m=+2354.830801431" watchObservedRunningTime="2026-11-02 21:48:35.283508589 +0000 UTC m=+2354.831758390" Nov 02 21:48:46.059834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:46.059805 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:48:46.060321 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:48:46.060007 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:48:55.944070 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:48:55.944030 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:49:08.607257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:08.607222 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:49:08.607694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:08.607471 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" podUID="984c32f6-6968-45db-bfac-a92667868c08" containerName="maas-api" containerID="cri-o://9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f" gracePeriod=30 Nov 02 21:49:08.896316 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:08.896290 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:49:09.056493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.056457 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\") pod \"984c32f6-6968-45db-bfac-a92667868c08\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " Nov 02 21:49:09.056684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.056603 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\") pod \"984c32f6-6968-45db-bfac-a92667868c08\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " Nov 02 21:49:09.056684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.056663 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\" (UniqueName: \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\") pod \"984c32f6-6968-45db-bfac-a92667868c08\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " Nov 02 21:49:09.056804 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.056690 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") pod \"984c32f6-6968-45db-bfac-a92667868c08\" (UID: \"984c32f6-6968-45db-bfac-a92667868c08\") " Nov 02 21:49:09.056861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.056794 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca" pod "984c32f6-6968-45db-bfac-a92667868c08" (UID: "984c32f6-6968-45db-bfac-a92667868c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:09.057193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.057173 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/984c32f6-6968-45db-bfac-a92667868c08-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:09.058787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.058764 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6" pod "984c32f6-6968-45db-bfac-a92667868c08" (UID: "984c32f6-6968-45db-bfac-a92667868c08"). InnerVolumeSpecName "kube-api-access-78xq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:09.058890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.058773 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs" pod "984c32f6-6968-45db-bfac-a92667868c08" (UID: "984c32f6-6968-45db-bfac-a92667868c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:09.058890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.058814 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls" pod "984c32f6-6968-45db-bfac-a92667868c08" (UID: "984c32f6-6968-45db-bfac-a92667868c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:09.158519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.158463 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:09.158519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.158483 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xq6\" (UniqueName: \"kubernetes.io/projected/984c32f6-6968-45db-bfac-a92667868c08-kube-api-access-78xq6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:09.158519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.158494 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/984c32f6-6968-45db-bfac-a92667868c08-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:09.258716 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.258680 5859 generic.go:356] "Generic (PLEG): container finished" podID="984c32f6-6968-45db-bfac-a92667868c08" containerID="9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f" exitCode=0 Nov 02 21:49:09.258888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.258721 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" Nov 02 21:49:09.258888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.258766 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" event={"ID":"984c32f6-6968-45db-bfac-a92667868c08","Type":"ContainerDied","Data":"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f"} Nov 02 21:49:09.258888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.258824 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd" event={"ID":"984c32f6-6968-45db-bfac-a92667868c08","Type":"ContainerDied","Data":"675219e7c6fd826a8fddbc11948e240cb77d9b173ef1d845145bd583c80517bd"} Nov 02 21:49:09.258888 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.258857 5859 scope.go:122] "RemoveContainer" containerID="9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f" Nov 02 21:49:09.269351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.269330 5859 scope.go:122] "RemoveContainer" containerID="9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f" Nov 02 21:49:09.269662 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:09.269635 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f\": container with ID starting with 9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f not found: ID does not exist" containerID="9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f" Nov 02 21:49:09.269772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.269672 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f"} err="failed to get container status \"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f\": rpc error: code = NotFound desc = could not find container \"9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f\": container with ID starting with 9ec2bd7d79322e18c03197fba4042d3a0a87b208c952a1bd55f4dadb3308d36f not found: ID does not exist" Nov 02 21:49:09.423816 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.423753 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:49:09.426105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.426085 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3997-6b874c45f7-zpqsd"] Nov 02 21:49:09.866478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.866445 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:49:09.866938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.866669 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerName="maas-api" containerID="cri-o://a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d" gracePeriod=30 Nov 02 21:49:09.991725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:09.991694 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.115:8443/health\": dial tcp 10.134.0.115:8443: connect: connection refused" Nov 02 21:49:10.156242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.156216 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:49:10.202849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.202819 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") pod \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " Nov 02 21:49:10.203008 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.202988 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\" (UniqueName: \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\") pod \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " Nov 02 21:49:10.203080 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.203054 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\") pod \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " Nov 02 21:49:10.203160 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.203144 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\") pod \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\" (UID: \"1b7bc367-c644-4422-a5bf-7b1f5227d73a\") " Nov 02 21:49:10.203393 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.203359 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca" pod "1b7bc367-c644-4422-a5bf-7b1f5227d73a" (UID: "1b7bc367-c644-4422-a5bf-7b1f5227d73a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:10.203708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.203686 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7bc367-c644-4422-a5bf-7b1f5227d73a-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:10.205083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.205056 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs" pod "1b7bc367-c644-4422-a5bf-7b1f5227d73a" (UID: "1b7bc367-c644-4422-a5bf-7b1f5227d73a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:10.205196 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.205042 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6" pod "1b7bc367-c644-4422-a5bf-7b1f5227d73a" (UID: "1b7bc367-c644-4422-a5bf-7b1f5227d73a"). InnerVolumeSpecName "kube-api-access-whvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:10.205454 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.205435 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls" pod "1b7bc367-c644-4422-a5bf-7b1f5227d73a" (UID: "1b7bc367-c644-4422-a5bf-7b1f5227d73a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:10.267088 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.267060 5859 generic.go:356] "Generic (PLEG): container finished" podID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerID="a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d" exitCode=0 Nov 02 21:49:10.267190 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.267108 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" Nov 02 21:49:10.267190 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.267125 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" event={"ID":"1b7bc367-c644-4422-a5bf-7b1f5227d73a","Type":"ContainerDied","Data":"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d"} Nov 02 21:49:10.267264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.267201 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5" event={"ID":"1b7bc367-c644-4422-a5bf-7b1f5227d73a","Type":"ContainerDied","Data":"bf98b4946febc32d777cf4dc60ec541592d6aa5dac9d7a3de6f9007884ffc2f0"} Nov 02 21:49:10.267264 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.267228 5859 scope.go:122] "RemoveContainer" containerID="a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d" Nov 02 21:49:10.281206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.281189 5859 scope.go:122] "RemoveContainer" containerID="a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d" Nov 02 21:49:10.281572 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:10.281537 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d\": container with ID starting with a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d not found: ID does not exist" containerID="a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d" Nov 02 21:49:10.281674 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.281575 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d"} err="failed to get container status \"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d\": rpc error: code = NotFound desc = could not find container \"a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d\": container with ID starting with a50ee130e74b7f91de67389ac624f16a0da79cfc7a4ce63379aaa51cbc2d340d not found: ID does not exist" Nov 02 21:49:10.304773 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.304748 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvx6\" (UniqueName: \"kubernetes.io/projected/1b7bc367-c644-4422-a5bf-7b1f5227d73a-kube-api-access-whvx6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:10.304870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.304780 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:10.304870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.304795 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7bc367-c644-4422-a5bf-7b1f5227d73a-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:10.449105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.449027 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:49:10.453413 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.453391 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa60bc-695dc8c674-x9rg5"] Nov 02 21:49:10.570695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.570664 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" path="/var/lib/kubelet/pods/1b7bc367-c644-4422-a5bf-7b1f5227d73a/volumes" Nov 02 21:49:10.571225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:10.571200 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984c32f6-6968-45db-bfac-a92667868c08" path="/var/lib/kubelet/pods/984c32f6-6968-45db-bfac-a92667868c08/volumes" Nov 02 21:49:20.193828 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:20.193796 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:49:25.194246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.194189 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" podUID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" containerName="payload-processing" containerID="cri-o://edc736c0b2e166da291be07d7b20d7619cae1ef37073388f8fa338d9c1c04f46" gracePeriod=25 Nov 02 21:49:25.366099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.365941 5859 generic.go:356] "Generic (PLEG): container finished" podID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" containerID="edc736c0b2e166da291be07d7b20d7619cae1ef37073388f8fa338d9c1c04f46" exitCode=0 Nov 02 21:49:25.366099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.366065 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" event={"ID":"9338a6d1-f91b-41fb-a19e-d347233a8d1a","Type":"ContainerDied","Data":"edc736c0b2e166da291be07d7b20d7619cae1ef37073388f8fa338d9c1c04f46"} Nov 02 21:49:25.455822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.455770 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:49:25.560703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.560673 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\" (UniqueName: \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\") pod \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " Nov 02 21:49:25.560854 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.560780 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\") pod \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\" (UID: \"9338a6d1-f91b-41fb-a19e-d347233a8d1a\") " Nov 02 21:49:25.561193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.561169 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume" pod "9338a6d1-f91b-41fb-a19e-d347233a8d1a" (UID: "9338a6d1-f91b-41fb-a19e-d347233a8d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:25.562705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.562680 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd" pod "9338a6d1-f91b-41fb-a19e-d347233a8d1a" (UID: "9338a6d1-f91b-41fb-a19e-d347233a8d1a"). InnerVolumeSpecName "kube-api-access-zftsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:25.662939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.662909 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zftsd\" (UniqueName: \"kubernetes.io/projected/9338a6d1-f91b-41fb-a19e-d347233a8d1a-kube-api-access-zftsd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:25.662939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:25.662935 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9338a6d1-f91b-41fb-a19e-d347233a8d1a-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:25.947692 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:25.947659 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:49:26.373937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.373903 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" event={"ID":"9338a6d1-f91b-41fb-a19e-d347233a8d1a","Type":"ContainerDied","Data":"80d0229eff469eeec2c4e58bca015278e9f52d0c9386da49c41115e1e5b83022"} Nov 02 21:49:26.373937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.373933 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d" Nov 02 21:49:26.374453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.373987 5859 scope.go:122] "RemoveContainer" containerID="edc736c0b2e166da291be07d7b20d7619cae1ef37073388f8fa338d9c1c04f46" Nov 02 21:49:26.538678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.538643 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:49:26.541860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.541834 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cc0d6a-85fbd5d65f-xtz8d"] Nov 02 21:49:26.571198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:26.571171 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" path="/var/lib/kubelet/pods/9338a6d1-f91b-41fb-a19e-d347233a8d1a/volumes" Nov 02 21:49:28.385628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.385596 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:49:28.386424 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386405 5859 cpu_manager.go:414] "containerMap: removing container" podUID="984c32f6-6968-45db-bfac-a92667868c08" containerName="maas-api" Nov 02 21:49:28.386479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386426 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984c32f6-6968-45db-bfac-a92667868c08" containerName="maas-api" Nov 02 21:49:28.386479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386464 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerName="maas-api" Nov 02 21:49:28.386479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386472 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerName="maas-api" Nov 02 21:49:28.386570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386502 5859 cpu_manager.go:414] "containerMap: removing container" podUID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" containerName="payload-processing" Nov 02 21:49:28.386570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386510 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" containerName="payload-processing" Nov 02 21:49:28.386657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386646 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9338a6d1-f91b-41fb-a19e-d347233a8d1a" containerName="payload-processing" Nov 02 21:49:28.386692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386665 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="984c32f6-6968-45db-bfac-a92667868c08" containerName="maas-api" Nov 02 21:49:28.386692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.386680 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7bc367-c644-4422-a5bf-7b1f5227d73a" containerName="maas-api" Nov 02 21:49:28.391497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.391480 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.446298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.446267 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:49:28.490549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.490524 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bb45bc-serving-cert\"" Nov 02 21:49:28.491273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.491251 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzs5\" (UniqueName: \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.491371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.491287 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.491371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.491315 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.491371 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.491365 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.592711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.592680 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzs5\" (UniqueName: \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.592860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.592718 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.592860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.592742 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.592860 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:28.592817 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-serving-cert: secret "maas-api-e2e-praxis-b-bb45bc-serving-cert" not found Nov 02 21:49:28.592860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.592826 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.593062 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:28.592884 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls podName:6eef6c43-6199-4e5c-9f26-5ccac44e59d5 nodeName:}" failed. No retries permitted until 2026-11-02 21:49:29.092865218 +0000 UTC m=+2408.641114980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls") pod "maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" (UID: "6eef6c43-6199-4e5c-9f26-5ccac44e59d5") : secret "maas-api-e2e-praxis-b-bb45bc-serving-cert" not found Nov 02 21:49:28.593364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.593345 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.595239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.595221 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:28.651058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:28.650972 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzs5\" (UniqueName: \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:29.099324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:29.099293 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:29.101461 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:29.101441 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:29.304908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:29.304877 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:29.678588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:29.678562 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:49:29.680368 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:49:29.680339 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eef6c43_6199_4e5c_9f26_5ccac44e59d5.slice/crio-5ba130287c1e2bacbaef6a0d6dec2f7e579e7ad45c3698f29e4b528d683d8984 WatchSource:0}: Error finding container 5ba130287c1e2bacbaef6a0d6dec2f7e579e7ad45c3698f29e4b528d683d8984: Status 404 returned error can't find the container with id 5ba130287c1e2bacbaef6a0d6dec2f7e579e7ad45c3698f29e4b528d683d8984 Nov 02 21:49:30.402356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:30.402324 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" event={"ID":"6eef6c43-6199-4e5c-9f26-5ccac44e59d5","Type":"ContainerStarted","Data":"5ba130287c1e2bacbaef6a0d6dec2f7e579e7ad45c3698f29e4b528d683d8984"} Nov 02 21:49:31.409790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:31.409756 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" event={"ID":"6eef6c43-6199-4e5c-9f26-5ccac44e59d5","Type":"ContainerStarted","Data":"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb"} Nov 02 21:49:31.410208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:31.409823 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:31.526789 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:31.526739 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" podStartSLOduration=2.624206489 podStartE2EDuration="3.52672381s" totalImagesPullingTime="902.517321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:31.523941016 +0000 UTC m=+2411.072190806" watchObservedRunningTime="2026-11-02 21:49:31.52672381 +0000 UTC m=+2411.074973591" Nov 02 21:49:37.421913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:49:37.421877 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:49:55.945174 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:49:55.945140 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:50:01.840744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:01.840661 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:50:01.841348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:01.840986 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" containerID="cri-o://980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d" gracePeriod=30 Nov 02 21:50:01.943585 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:01.943553 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb412a8_f9df_43b5_9416_a0ce1b96a96f.slice/crio-conmon-980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:50:02.164083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.164057 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:50:02.273462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273429 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273471 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273562 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273614 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273661 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273686 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.273822 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.273742 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\") pod \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\" (UID: \"7eb412a8-f9df-43b5-9416-a0ce1b96a96f\") " Nov 02 21:50:02.274490 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.274464 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:02.276218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276185 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:02.276218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276197 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:02.276382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276305 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:02.276382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276338 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "kube-api-access-nxbhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:02.276473 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276456 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:02.276509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.276484 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts" pod "7eb412a8-f9df-43b5-9416-a0ce1b96a96f" (UID: "7eb412a8-f9df-43b5-9416-a0ce1b96a96f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:02.375144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375117 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-dns-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375144 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375142 5859 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-kubeconfig\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375152 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbhs\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-kube-api-access-nxbhs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375162 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375170 5859 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-local-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375180 5859 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-cacerts\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.375331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.375188 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7eb412a8-f9df-43b5-9416-a0ce1b96a96f-istio-csr-ca-configmap\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:02.599120 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.599090 5859 generic.go:356] "Generic (PLEG): container finished" podID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerID="980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d" exitCode=0 Nov 02 21:50:02.599244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.599136 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" Nov 02 21:50:02.599244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.599138 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" event={"ID":"7eb412a8-f9df-43b5-9416-a0ce1b96a96f","Type":"ContainerDied","Data":"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d"} Nov 02 21:50:02.599244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.599189 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k" event={"ID":"7eb412a8-f9df-43b5-9416-a0ce1b96a96f","Type":"ContainerDied","Data":"0ea8c738979fea0b178b229f2b532583088b0af2b33ab1693b7828e221875737"} Nov 02 21:50:02.599244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.599206 5859 scope.go:122] "RemoveContainer" containerID="980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d" Nov 02 21:50:02.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.634047 5859 scope.go:122] "RemoveContainer" containerID="980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d" Nov 02 21:50:02.634333 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:02.634308 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d\": container with ID starting with 980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d not found: ID does not exist" containerID="980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d" Nov 02 21:50:02.634382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.634343 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d"} err="failed to get container status \"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d\": rpc error: code = NotFound desc = could not find container \"980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d\": container with ID starting with 980f15ff52ce85f2cb0daef34ac5e7e66e35bc6c76b8dcdc9660d94a1a5daf2d not found: ID does not exist" Nov 02 21:50:02.766477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.766444 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:50:02.830411 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:02.830335 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8x99k"] Nov 02 21:50:04.571771 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:04.571731 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" path="/var/lib/kubelet/pods/7eb412a8-f9df-43b5-9416-a0ce1b96a96f/volumes" Nov 02 21:50:08.728531 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:08.728504 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@11-35093-0.service\": RecentStats: unable to find data in memory cache]" Nov 02 21:50:09.679532 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.679496 5859 generic.go:356] "Generic (PLEG): container finished" podID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerID="5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec" exitCode=0 Nov 02 21:50:09.679721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.679569 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerDied","Data":"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec"} Nov 02 21:50:09.679977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.679937 5859 scope.go:122] "RemoveContainer" containerID="5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec" Nov 02 21:50:09.681227 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.681199 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:09.681303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.681267 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:09.681303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:09.681299 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:10.687818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:10.687785 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerStarted","Data":"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc"} Nov 02 21:50:11.077530 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:11.077506 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:50:11.078848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:11.078813 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:50:11.079023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:11.078871 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 21:50:12.078482 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:12.078444 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:50:12.078878 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:12.078507 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 21:50:13.017984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:13.017032 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:50:13.077713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:13.077676 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:50:13.077881 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:13.077735 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 21:50:14.078448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:14.078402 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:50:14.078933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:14.078467 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 21:50:15.077535 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:15.077498 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 02 21:50:15.077713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:15.077555 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 02 21:50:16.082961 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:16.082918 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:50:16.084091 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:16.084069 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:50:19.779995 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.779930 5859 generic.go:356] "Generic (PLEG): container finished" podID="03ee309f-b111-403b-9e19-846981c524d3" containerID="498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e" exitCode=0 Nov 02 21:50:19.780435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.780009 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerDied","Data":"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e"} Nov 02 21:50:19.780435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.780427 5859 scope.go:122] "RemoveContainer" containerID="498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e" Nov 02 21:50:19.781868 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.781844 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:19.781934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.781907 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:19.781994 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:19.781939 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:50:20.788772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:20.788736 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerStarted","Data":"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969"} Nov 02 21:50:21.276790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:21.276758 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:50:21.277130 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:21.277100 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:50:21.277205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:21.277181 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:50:22.277440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:22.277397 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:50:22.277916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:22.277459 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:50:23.276612 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:23.276580 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:50:23.276776 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:23.276632 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:50:24.277018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:24.276984 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:50:24.277403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:24.277039 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:50:25.277058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:25.277010 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:50:25.277540 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:25.277093 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:50:25.955793 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:25.955756 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:50:26.302630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:26.302554 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:50:26.303252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:26.303220 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:50:26.304023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:26.304007 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:50:27.793244 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.793214 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:50:27.793744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.793732 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" Nov 02 21:50:27.793788 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.793745 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" Nov 02 21:50:27.793883 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.793874 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb412a8-f9df-43b5-9416-a0ce1b96a96f" containerName="discovery" Nov 02 21:50:27.798688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.798670 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:27.851252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.851234 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eabda2-serving-cert\"" Nov 02 21:50:27.904162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.904133 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:50:27.930700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.930667 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbtr\" (UniqueName: \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:27.930825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.930711 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:27.930825 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.930747 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:27.930933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:27.930892 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.031853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.031822 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbtr\" (UniqueName: \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.031853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.031856 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.032099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.031898 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.032099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.031926 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.032099 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:28.031971 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-serving-cert: secret "maas-api-e2e-praxis-a-eabda2-serving-cert" not found Nov 02 21:50:28.032099 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:28.032032 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls podName:945da409-b927-4d49-94cf-83a4655f5b44 nodeName:}" failed. No retries permitted until 2026-11-02 21:50:28.532018578 +0000 UTC m=+2468.080268340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls") pod "maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" (UID: "945da409-b927-4d49-94cf-83a4655f5b44") : secret "maas-api-e2e-praxis-a-eabda2-serving-cert" not found Nov 02 21:50:28.032539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.032516 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.034262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.034242 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.089561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.089527 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbtr\" (UniqueName: \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.538927 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.538849 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.541266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.541240 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:28.713403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:28.713371 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:29.087374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:29.087345 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:50:29.088099 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:50:29.088075 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945da409_b927_4d49_94cf_83a4655f5b44.slice/crio-d57b54ea78792a565017ff7e18dca9a3715bd8f3a9c76ced742137b27a34639b WatchSource:0}: Error finding container d57b54ea78792a565017ff7e18dca9a3715bd8f3a9c76ced742137b27a34639b: Status 404 returned error can't find the container with id d57b54ea78792a565017ff7e18dca9a3715bd8f3a9c76ced742137b27a34639b Nov 02 21:50:29.869736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:29.869703 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" event={"ID":"945da409-b927-4d49-94cf-83a4655f5b44","Type":"ContainerStarted","Data":"d57b54ea78792a565017ff7e18dca9a3715bd8f3a9c76ced742137b27a34639b"} Nov 02 21:50:30.877573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:30.877538 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" event={"ID":"945da409-b927-4d49-94cf-83a4655f5b44","Type":"ContainerStarted","Data":"f8d889564238077a58ffdafd4a55e37fd13473e5b7843b71f66d1768da856dbf"} Nov 02 21:50:30.878061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:30.877627 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:30.995306 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:30.995252 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" podStartSLOduration=2.978376658 podStartE2EDuration="3.995239498s" totalImagesPullingTime="1.01686284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:30.99294163 +0000 UTC m=+2470.541191412" watchObservedRunningTime="2026-11-02 21:50:30.995239498 +0000 UTC m=+2470.543489278" Nov 02 21:50:36.889923 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:36.889891 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:50:55.947039 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:55.947000 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:50:58.444795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.444761 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:50:58.445279 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.445016 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" podUID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" containerName="maas-api" containerID="cri-o://50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f" gracePeriod=30 Nov 02 21:50:58.718861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.718838 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:50:58.783783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.783761 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\" (UniqueName: \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\") pod \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " Nov 02 21:50:58.783921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.783825 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\") pod \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " Nov 02 21:50:58.783921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.783913 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") pod \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " Nov 02 21:50:58.784032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.783998 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\") pod \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\" (UID: \"b1a719b0-bead-4e9b-9b2e-e5a5f7121048\") " Nov 02 21:50:58.784247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.784220 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca" pod "b1a719b0-bead-4e9b-9b2e-e5a5f7121048" (UID: "b1a719b0-bead-4e9b-9b2e-e5a5f7121048"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:58.784556 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.784534 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:58.785942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.785907 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs" pod "b1a719b0-bead-4e9b-9b2e-e5a5f7121048" (UID: "b1a719b0-bead-4e9b-9b2e-e5a5f7121048"). InnerVolumeSpecName "kube-api-access-m4dgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:58.786069 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.785989 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs" pod "b1a719b0-bead-4e9b-9b2e-e5a5f7121048" (UID: "b1a719b0-bead-4e9b-9b2e-e5a5f7121048"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:58.786124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.786080 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls" pod "b1a719b0-bead-4e9b-9b2e-e5a5f7121048" (UID: "b1a719b0-bead-4e9b-9b2e-e5a5f7121048"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:58.885748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.885715 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:58.885748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.885747 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:58.885980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:58.885759 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dgs\" (UniqueName: \"kubernetes.io/projected/b1a719b0-bead-4e9b-9b2e-e5a5f7121048-kube-api-access-m4dgs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:59.059164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.059085 5859 generic.go:356] "Generic (PLEG): container finished" podID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" containerID="50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f" exitCode=0 Nov 02 21:50:59.059164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.059133 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" Nov 02 21:50:59.059164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.059160 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" event={"ID":"b1a719b0-bead-4e9b-9b2e-e5a5f7121048","Type":"ContainerDied","Data":"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f"} Nov 02 21:50:59.059409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.059200 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9" event={"ID":"b1a719b0-bead-4e9b-9b2e-e5a5f7121048","Type":"ContainerDied","Data":"59cefe8d5f4ef7c7eb93a4a451b9a15bd53996e6d3d0c808caba6407c0c61b68"} Nov 02 21:50:59.059409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.059220 5859 scope.go:122] "RemoveContainer" containerID="50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f" Nov 02 21:50:59.069984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.069961 5859 scope.go:122] "RemoveContainer" containerID="50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f" Nov 02 21:50:59.070242 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:50:59.070226 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f\": container with ID starting with 50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f not found: ID does not exist" containerID="50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f" Nov 02 21:50:59.070289 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.070249 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f"} err="failed to get container status \"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f\": rpc error: code = NotFound desc = could not find container \"50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f\": container with ID starting with 50bdf8dc8eb819e63d9f7cb641b49192cb157b15a86cc1412839c5ca130e6f6f not found: ID does not exist" Nov 02 21:50:59.224303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.224271 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:50:59.227441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:50:59.227421 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8e90da-7d5d9556c7-6bzc9"] Nov 02 21:51:00.570627 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:00.570597 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" path="/var/lib/kubelet/pods/b1a719b0-bead-4e9b-9b2e-e5a5f7121048/volumes" Nov 02 21:51:01.106998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.106970 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:51:01.107547 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.107533 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" containerName="maas-api" Nov 02 21:51:01.107601 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.107549 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" containerName="maas-api" Nov 02 21:51:01.107647 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.107638 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a719b0-bead-4e9b-9b2e-e5a5f7121048" containerName="maas-api" Nov 02 21:51:01.111678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.111662 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210469 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210515 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210592 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210728 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210644 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210783 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210804 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210826 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gjm\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.210924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.210847 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.211066 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.211041 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.217133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.217113 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:51:01.312003 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.311944 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312019 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312056 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312102 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312242 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312271 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312307 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gjm\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312314 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312343 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312441 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312495 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312617 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312719 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312679 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.312963 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.312935 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.314622 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.314597 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.314716 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.314700 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.369559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.369508 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.369713 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.369662 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gjm\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.425324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.425303 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:01.850433 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.850403 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:51:01.851279 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:51:01.851251 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b8fafd_3e6b_4a59_9836_a022f9f1be8b.slice/crio-0bfe855fda80f55138de4247a4571b44afbad7a4b82b6bc00cc572135eb5193b WatchSource:0}: Error finding container 0bfe855fda80f55138de4247a4571b44afbad7a4b82b6bc00cc572135eb5193b: Status 404 returned error can't find the container with id 0bfe855fda80f55138de4247a4571b44afbad7a4b82b6bc00cc572135eb5193b Nov 02 21:51:01.853434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.853403 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:01.853529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.853472 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:01.853529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:01.853521 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:02.082501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.082463 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" event={"ID":"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b","Type":"ContainerStarted","Data":"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d"} Nov 02 21:51:02.082669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.082521 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" event={"ID":"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b","Type":"ContainerStarted","Data":"0bfe855fda80f55138de4247a4571b44afbad7a4b82b6bc00cc572135eb5193b"} Nov 02 21:51:02.202924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.202818 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podStartSLOduration=1.202800667 podStartE2EDuration="1.202800667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:02.199556368 +0000 UTC m=+2501.747806169" watchObservedRunningTime="2026-11-02 21:51:02.202800667 +0000 UTC m=+2501.751050453" Nov 02 21:51:02.429979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.429020 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:02.431597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.431559 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:02.431748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:02.431615 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:03.427338 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:03.427301 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:03.428269 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:03.427862 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:04.430342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:04.430306 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:04.430707 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:04.430368 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:05.426294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:05.426257 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:05.426613 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:05.426572 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:06.429977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:06.429480 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:06.429977 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:06.429558 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:07.426996 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:07.426202 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:07.426996 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:07.426259 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:08.426497 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:08.426459 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:51:08.426879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:08.426533 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:51:09.052803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.052774 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:51:09.057106 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.057085 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.114343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.114313 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:51:09.160645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.160621 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-aa8aff-openshift-default-dockercfg-dwvhh\"" Nov 02 21:51:09.212479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212455 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212505 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212576 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212686 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212674 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212699 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212732 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212838 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212873 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9v9\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.212930 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.212900 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.313649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313589 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.313760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313652 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.313760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313671 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.313760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313700 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.313760 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313738 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313769 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9v9\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313798 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313837 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.313900 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314032 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.314002 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314276 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.314117 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.314457 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.314475 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.314618 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.314611 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.316432 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.316411 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.316529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.316435 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.369999 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.369969 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.370123 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.370108 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9v9\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\") pod \"e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.370781 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.370765 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:09.430847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.430820 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:09.432477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.432160 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:09.434204 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.432817 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:51:09.782423 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.782397 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:51:09.783234 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:51:09.783211 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8687284_6702_4573_a8d6_c5b5464ab9f6.slice/crio-7a575e78a990c474ac312a62d28884a88f911fc4ff5bcd8b26e75a5d9e04db72 WatchSource:0}: Error finding container 7a575e78a990c474ac312a62d28884a88f911fc4ff5bcd8b26e75a5d9e04db72: Status 404 returned error can't find the container with id 7a575e78a990c474ac312a62d28884a88f911fc4ff5bcd8b26e75a5d9e04db72 Nov 02 21:51:09.785666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.785631 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:09.785758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.785707 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:09.785758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:09.785749 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:10.155867 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.155828 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" event={"ID":"e8687284-6702-4573-a8d6-c5b5464ab9f6","Type":"ContainerStarted","Data":"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c"} Nov 02 21:51:10.156053 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.155889 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" event={"ID":"e8687284-6702-4573-a8d6-c5b5464ab9f6","Type":"ContainerStarted","Data":"7a575e78a990c474ac312a62d28884a88f911fc4ff5bcd8b26e75a5d9e04db72"} Nov 02 21:51:10.273626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.273565 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podStartSLOduration=1.273550133 podStartE2EDuration="1.273550133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:10.271450487 +0000 UTC m=+2509.819700295" watchObservedRunningTime="2026-11-02 21:51:10.273550133 +0000 UTC m=+2509.821799913" Nov 02 21:51:10.371562 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.371533 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:10.373239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.373213 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:10.373370 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:10.373256 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:11.372297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:11.372248 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:11.372830 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:11.372353 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:12.371726 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:12.371684 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:12.372213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:12.372167 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:13.372192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:13.372149 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:13.372629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:13.372210 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:14.371278 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:14.371240 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:14.371473 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:14.371293 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:15.372058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:15.372018 5859 patch_prober.go:29] interesting pod/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 21:51:15.372474 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:15.372093 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 21:51:15.856638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:15.856605 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:51:16.375718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:16.375686 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:16.376257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:16.376037 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:16.376765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:16.376744 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:51:20.857902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:20.857867 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" podUID="ad17068f-1f5a-4004-8bdf-303d57860eb4" containerName="payload-pre-processing" containerID="cri-o://546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6" gracePeriod=25 Nov 02 21:51:21.107342 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.107318 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:51:21.218391 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.218319 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\" (UniqueName: \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\") pod \"ad17068f-1f5a-4004-8bdf-303d57860eb4\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " Nov 02 21:51:21.218529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.218490 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\") pod \"ad17068f-1f5a-4004-8bdf-303d57860eb4\" (UID: \"ad17068f-1f5a-4004-8bdf-303d57860eb4\") " Nov 02 21:51:21.218823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.218799 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume" pod "ad17068f-1f5a-4004-8bdf-303d57860eb4" (UID: "ad17068f-1f5a-4004-8bdf-303d57860eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:21.218934 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.218920 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad17068f-1f5a-4004-8bdf-303d57860eb4-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:21.220420 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.220400 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988" pod "ad17068f-1f5a-4004-8bdf-303d57860eb4" (UID: "ad17068f-1f5a-4004-8bdf-303d57860eb4"). InnerVolumeSpecName "kube-api-access-5w988". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:21.270438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.270412 5859 generic.go:356] "Generic (PLEG): container finished" podID="ad17068f-1f5a-4004-8bdf-303d57860eb4" containerID="546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6" exitCode=0 Nov 02 21:51:21.270581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.270456 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" Nov 02 21:51:21.270581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.270504 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" event={"ID":"ad17068f-1f5a-4004-8bdf-303d57860eb4","Type":"ContainerDied","Data":"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6"} Nov 02 21:51:21.270581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.270556 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh" event={"ID":"ad17068f-1f5a-4004-8bdf-303d57860eb4","Type":"ContainerDied","Data":"7f72c270d8934dd6d8f5331c0a54fc764d253c8b53c63cda5b618f1392236313"} Nov 02 21:51:21.270756 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.270583 5859 scope.go:122] "RemoveContainer" containerID="546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6" Nov 02 21:51:21.281189 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.281168 5859 scope.go:122] "RemoveContainer" containerID="546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6" Nov 02 21:51:21.281444 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:51:21.281426 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6\": container with ID starting with 546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6 not found: ID does not exist" containerID="546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6" Nov 02 21:51:21.281489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.281452 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6"} err="failed to get container status \"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6\": rpc error: code = NotFound desc = could not find container \"546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6\": container with ID starting with 546f75beabf195d472f78f25cae91a2a6be525a821733c6f8b3ae335206861e6 not found: ID does not exist" Nov 02 21:51:21.320331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.320311 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w988\" (UniqueName: \"kubernetes.io/projected/ad17068f-1f5a-4004-8bdf-303d57860eb4-kube-api-access-5w988\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:21.436212 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.436185 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:51:21.442318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:21.442297 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-942164-7455986575-xl5mh"] Nov 02 21:51:22.570294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:22.570257 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad17068f-1f5a-4004-8bdf-303d57860eb4" path="/var/lib/kubelet/pods/ad17068f-1f5a-4004-8bdf-303d57860eb4/volumes" Nov 02 21:51:25.944931 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:51:25.944889 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:51:31.156062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.146290 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:51:31.156062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.147301 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ad17068f-1f5a-4004-8bdf-303d57860eb4" containerName="payload-pre-processing" Nov 02 21:51:31.156062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.147324 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad17068f-1f5a-4004-8bdf-303d57860eb4" containerName="payload-pre-processing" Nov 02 21:51:31.156062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.147488 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad17068f-1f5a-4004-8bdf-303d57860eb4" containerName="payload-pre-processing" Nov 02 21:51:31.156062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.154784 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.241841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.241802 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.251856 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.251925 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.251967 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252024 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252053 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252096 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252157 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252217 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zwh\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.252614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.252364 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353479 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353447 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353495 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353526 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353541 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353573 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353659 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353645 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353946 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353705 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353946 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353762 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.353946 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.353825 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zwh\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.354163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.354086 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.354163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.354109 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.354270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.354257 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.354325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.354287 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.354325 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.354294 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.356140 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.356112 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.356252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.356112 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.422186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.422134 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zwh\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.423049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.423026 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.470940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.470909 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:31.898243 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.898216 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:51:31.899479 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:51:31.899452 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3edf2e_1ee7_48dc_90d9_c8895bc0212c.slice/crio-cbc4771a040287934bf2bda4868c588b957293b5bd619530a65f26ccfb50e200 WatchSource:0}: Error finding container cbc4771a040287934bf2bda4868c588b957293b5bd619530a65f26ccfb50e200: Status 404 returned error can't find the container with id cbc4771a040287934bf2bda4868c588b957293b5bd619530a65f26ccfb50e200 Nov 02 21:51:31.901594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.901563 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:31.901691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.901620 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:31.901691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:31.901646 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:32.339417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.339379 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" event={"ID":"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c","Type":"ContainerStarted","Data":"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1"} Nov 02 21:51:32.339898 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.339433 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" event={"ID":"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c","Type":"ContainerStarted","Data":"cbc4771a040287934bf2bda4868c588b957293b5bd619530a65f26ccfb50e200"} Nov 02 21:51:32.458587 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.458533 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podStartSLOduration=1.458519068 podStartE2EDuration="1.458519068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:32.45629698 +0000 UTC m=+2532.004546761" watchObservedRunningTime="2026-11-02 21:51:32.458519068 +0000 UTC m=+2532.006768848" Nov 02 21:51:32.471425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.471395 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:32.473137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.473112 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:32.473257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:32.473173 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:33.472124 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:33.472087 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:33.472507 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:33.472152 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:34.471722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:34.471684 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:34.471899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:34.471734 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:35.471564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:35.471530 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:35.471921 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:35.471577 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:36.472368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:36.472327 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:36.472788 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:36.472383 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:37.471699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:37.471667 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:37.471862 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:37.471718 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:38.471415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:38.471380 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 02 21:51:38.471902 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:38.471431 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 02 21:51:39.475453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:39.475424 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:39.475880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:39.475812 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:39.476534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:51:39.476514 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:51:55.945606 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:51:55.945561 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:52:01.149571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.149541 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 21:52:01.153694 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.153675 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.200197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.200172 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:52:01.204313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.204294 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.251364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251338 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251495 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251404 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251542 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251617 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251705 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251691 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251728 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251842 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251896 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251875 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.251928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.251909 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.281408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.281382 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:52:01.351193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.351164 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 21:52:01.352703 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.352679 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.352824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.352727 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.352877 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.352833 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.352998 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.352981 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353061 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353025 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353118 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353062 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353116 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sjd\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.353218 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353172 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353227 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353275 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353251 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.353375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353285 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.353375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353321 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.353375 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353358 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.353533 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353374 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353533 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353389 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353695 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353669 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.353938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353907 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.354083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.353945 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.354083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.354013 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.354083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.354016 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.354083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.354074 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.354353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.354104 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.354353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.354245 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.356211 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.356185 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.356472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.356456 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.409757 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.409694 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.410009 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.409990 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.454818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.454796 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.454937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.454829 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.454937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.454868 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sjd\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.454937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.454922 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.454945 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455015 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455161 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455049 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455229 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455259 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455330 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455362 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455769 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455805 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.455928 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.455831 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.457331 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.457313 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.457522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.457499 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.467581 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.467561 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:01.512677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.512639 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.512834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.512699 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sjd\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.517682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.517624 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:01.894320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.894292 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 21:52:01.894765 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:52:01.894739 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da78730_8bdf_4110_9e91_ce41d18448c6.slice/crio-c503a25fc012543bd8eb1d01f36961526b07de3c706e9311a648e15c1518edec WatchSource:0}: Error finding container c503a25fc012543bd8eb1d01f36961526b07de3c706e9311a648e15c1518edec: Status 404 returned error can't find the container with id c503a25fc012543bd8eb1d01f36961526b07de3c706e9311a648e15c1518edec Nov 02 21:52:01.897150 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.897107 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.897258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.897168 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.897258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.897195 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.945981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.945939 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:52:01.946705 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:52:01.946684 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4eb182_c5e9_400e_8581_5204f8b48ed2.slice/crio-5f75265d921bfd3ab8cb5cd92957cbe7d676cac39d38a9fe673e40ed017bf1c1 WatchSource:0}: Error finding container 5f75265d921bfd3ab8cb5cd92957cbe7d676cac39d38a9fe673e40ed017bf1c1: Status 404 returned error can't find the container with id 5f75265d921bfd3ab8cb5cd92957cbe7d676cac39d38a9fe673e40ed017bf1c1 Nov 02 21:52:01.949171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.949144 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.949252 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.949206 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.949291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:01.949250 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:02.552472 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.552434 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" event={"ID":"de4eb182-c5e9-400e-8581-5204f8b48ed2","Type":"ContainerStarted","Data":"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a"} Nov 02 21:52:02.553164 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.553137 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" event={"ID":"de4eb182-c5e9-400e-8581-5204f8b48ed2","Type":"ContainerStarted","Data":"5f75265d921bfd3ab8cb5cd92957cbe7d676cac39d38a9fe673e40ed017bf1c1"} Nov 02 21:52:02.556394 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.556366 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" event={"ID":"6da78730-8bdf-4110-9e91-ce41d18448c6","Type":"ContainerStarted","Data":"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa"} Nov 02 21:52:02.556602 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.556587 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" event={"ID":"6da78730-8bdf-4110-9e91-ce41d18448c6","Type":"ContainerStarted","Data":"c503a25fc012543bd8eb1d01f36961526b07de3c706e9311a648e15c1518edec"} Nov 02 21:52:02.683270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.683223 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podStartSLOduration=1.6832095 podStartE2EDuration="1.6832095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:02.682595052 +0000 UTC m=+2562.230844857" watchObservedRunningTime="2026-11-02 21:52:02.6832095 +0000 UTC m=+2562.231459275" Nov 02 21:52:02.874182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.874123 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podStartSLOduration=1.874104003 podStartE2EDuration="1.874104003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:02.827313759 +0000 UTC m=+2562.375563572" watchObservedRunningTime="2026-11-02 21:52:02.874104003 +0000 UTC m=+2562.422353785" Nov 02 21:52:02.877586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.877556 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:52:02.877824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:02.877798 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" containerID="cri-o://8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969" gracePeriod=30 Nov 02 21:52:03.468024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.467992 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:03.469386 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.469365 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:03.469447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.469411 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:03.520714 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.519046 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:03.525493 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.525457 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:03.525721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:03.525691 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:04.468861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:04.468827 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:04.469229 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:04.468920 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:04.518312 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:04.518279 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:04.518538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:04.518513 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:05.468440 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:05.468407 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:05.468589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:05.468465 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:05.524976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:05.524289 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:05.524976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:05.524361 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:06.468981 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:06.468922 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:06.469136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:06.469002 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:06.518976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:06.518323 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:06.518976 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:06.518380 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:07.468574 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:07.468532 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:07.469072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:07.468619 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:07.528975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:07.528703 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:07.528975 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:07.528756 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:08.152777 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.152716 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:52:08.205517 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205490 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205530 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205606 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205627 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205716 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205754 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.205829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205797 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:08.206012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205860 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.206012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205911 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:08.206012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205926 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:08.206012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.205947 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.206012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206001 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\") pod \"03ee309f-b111-403b-9e19-846981c524d3\" (UID: \"03ee309f-b111-403b-9e19-846981c524d3\") " Nov 02 21:52:08.206277 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206159 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:08.206629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206603 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.206730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206634 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.206730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206650 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.206730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.206664 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.207103 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.207065 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:08.208636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.208612 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:08.208740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.208635 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "kube-api-access-zblmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:08.208740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.208657 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:08.209291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.209266 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo" pod "03ee309f-b111-403b-9e19-846981c524d3" (UID: "03ee309f-b111-403b-9e19-846981c524d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:08.307712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.307670 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.307712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.307706 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee309f-b111-403b-9e19-846981c524d3-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.307712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.307719 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee309f-b111-403b-9e19-846981c524d3-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.307969 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.307731 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee309f-b111-403b-9e19-846981c524d3-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.307969 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.307746 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblmh\" (UniqueName: \"kubernetes.io/projected/03ee309f-b111-403b-9e19-846981c524d3-kube-api-access-zblmh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:08.468005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.467924 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 21:52:08.468005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.467994 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 21:52:08.519054 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.519026 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 21:52:08.519380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.519074 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 21:52:08.606374 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.606346 5859 generic.go:356] "Generic (PLEG): container finished" podID="03ee309f-b111-403b-9e19-846981c524d3" containerID="8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969" exitCode=0 Nov 02 21:52:08.606537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.606435 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" Nov 02 21:52:08.606537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.606434 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerDied","Data":"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969"} Nov 02 21:52:08.606537 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.606510 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk" event={"ID":"03ee309f-b111-403b-9e19-846981c524d3","Type":"ContainerDied","Data":"a58ffa8eb24507d6a3fe18630355830df16b628687ac37776bf1ff3da317e962"} Nov 02 21:52:08.606663 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.606543 5859 scope.go:122] "RemoveContainer" containerID="8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969" Nov 02 21:52:08.621250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.621227 5859 scope.go:122] "RemoveContainer" containerID="498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e" Nov 02 21:52:08.633539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.633503 5859 scope.go:122] "RemoveContainer" containerID="8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969" Nov 02 21:52:08.633872 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:52:08.633844 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969\": container with ID starting with 8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969 not found: ID does not exist" containerID="8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969" Nov 02 21:52:08.633991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.633884 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969"} err="failed to get container status \"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969\": rpc error: code = NotFound desc = could not find container \"8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969\": container with ID starting with 8e06ea0def9a6e945b8316278e1c06d303f3249a708279419944f5f14fc92969 not found: ID does not exist" Nov 02 21:52:08.633991 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.633909 5859 scope.go:122] "RemoveContainer" containerID="498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e" Nov 02 21:52:08.634304 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:52:08.634282 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e\": container with ID starting with 498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e not found: ID does not exist" containerID="498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e" Nov 02 21:52:08.634385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.634315 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e"} err="failed to get container status \"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e\": rpc error: code = NotFound desc = could not find container \"498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e\": container with ID starting with 498235c7ef4b84c2f107819f7807b65238a095680d67ec3a47f92ca435a2793e not found: ID does not exist" Nov 02 21:52:08.773348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.773260 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:52:08.777018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:08.776993 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-942164-openshift-default-5bb4cc8fc-sfqmk"] Nov 02 21:52:09.472351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.472322 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:09.472787 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.472764 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:09.473509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.473493 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 21:52:09.522023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.522005 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:09.522345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.522229 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:09.522886 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:09.522868 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:52:10.571485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:10.571453 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ee309f-b111-403b-9e19-846981c524d3" path="/var/lib/kubelet/pods/03ee309f-b111-403b-9e19-846981c524d3/volumes" Nov 02 21:52:25.575356 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.575327 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:52:25.576038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576017 5859 cpu_manager.go:414] "containerMap: removing container" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.576038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576036 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.576203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576052 5859 cpu_manager.go:414] "containerMap: removing container" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.576203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576057 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.576203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576175 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.576203 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.576184 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ee309f-b111-403b-9e19-846981c524d3" containerName="istio-proxy" Nov 02 21:52:25.581216 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.581196 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.634668 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.634638 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:52:25.681209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.681180 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3c7706-serving-cert\"" Nov 02 21:52:25.757909 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.757881 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.758099 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.757931 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.758167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.758148 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.758208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.758198 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69wb\" (UniqueName: \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.859775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.859675 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.859775 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.859754 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69wb\" (UniqueName: \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.860067 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.859898 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.860067 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.859944 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.860648 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.860624 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.862438 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.862414 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.862578 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.862557 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.917446 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:25.917424 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69wb\" (UniqueName: \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\") pod \"maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:25.943458 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:52:25.943419 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:52:26.197580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:26.197494 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:26.577026 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:26.577001 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:52:26.578248 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:52:26.578222 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c4e8e2_94ad_49eb_b9ef_5821cc905aa0.slice/crio-6e136827a4517deb6ac04254bec9627af84714916216d3494c97691adcd98737 WatchSource:0}: Error finding container 6e136827a4517deb6ac04254bec9627af84714916216d3494c97691adcd98737: Status 404 returned error can't find the container with id 6e136827a4517deb6ac04254bec9627af84714916216d3494c97691adcd98737 Nov 02 21:52:26.579385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:26.579367 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:52:26.735314 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:26.735287 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" event={"ID":"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0","Type":"ContainerStarted","Data":"6e136827a4517deb6ac04254bec9627af84714916216d3494c97691adcd98737"} Nov 02 21:52:27.744652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:27.744547 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" event={"ID":"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0","Type":"ContainerStarted","Data":"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035"} Nov 02 21:52:27.744652 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:27.744622 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:27.859847 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:27.859804 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" podStartSLOduration=1.971907854 podStartE2EDuration="2.859791823s" totalImagesPullingTime="887.883969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:27.858824393 +0000 UTC m=+2587.407074192" watchObservedRunningTime="2026-11-02 21:52:27.859791823 +0000 UTC m=+2587.408041632" Nov 02 21:52:33.755413 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:33.755385 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:52:41.770181 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:41.770133 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 21:52:55.565543 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.565507 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 21:52:55.569682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.569661 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.624505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.624482 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 21:52:55.670879 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.670861 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-97995b-dockercfg-p7c66\"" Nov 02 21:52:55.671018 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.670984 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-97995b\"" Nov 02 21:52:55.682637 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.682612 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlxm\" (UniqueName: \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.682770 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.682744 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.783586 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.783560 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlxm\" (UniqueName: \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.783711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.783610 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.784273 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.784255 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.841205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.841182 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlxm\" (UniqueName: \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\") pod \"payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.884195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:55.884174 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:55.966696 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:52:55.966659 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:52:56.268048 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:56.268020 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 21:52:56.269694 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:52:56.269670 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65af2a2_6481_4664_92d8_9f95ccc661f8.slice/crio-ee28ad128778cc2f8cc43c6d7cfa4c294130d8eba821e255bdf271f98cac19ff WatchSource:0}: Error finding container ee28ad128778cc2f8cc43c6d7cfa4c294130d8eba821e255bdf271f98cac19ff: Status 404 returned error can't find the container with id ee28ad128778cc2f8cc43c6d7cfa4c294130d8eba821e255bdf271f98cac19ff Nov 02 21:52:56.993250 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:56.993211 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" event={"ID":"f65af2a2-6481-4664-92d8-9f95ccc661f8","Type":"ContainerStarted","Data":"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060"} Nov 02 21:52:56.993667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:56.993263 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" event={"ID":"f65af2a2-6481-4664-92d8-9f95ccc661f8","Type":"ContainerStarted","Data":"ee28ad128778cc2f8cc43c6d7cfa4c294130d8eba821e255bdf271f98cac19ff"} Nov 02 21:52:56.993667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:56.993351 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:52:57.109599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:57.109546 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" podStartSLOduration=2.109530514 podStartE2EDuration="2.109530514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:57.106329194 +0000 UTC m=+2616.654578995" watchObservedRunningTime="2026-11-02 21:52:57.109530514 +0000 UTC m=+2616.657780297" Nov 02 21:52:58.297647 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.297616 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 21:52:58.301836 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.301820 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310377 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310353 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2xdm\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310392 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310419 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310564 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310529 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310641 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310625 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310664 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310765 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310814 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.310901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.310884 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.391940 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.391912 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6e535d-openshift-default-dockercfg-hg4dw\"" Nov 02 21:52:58.408103 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.408065 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 21:52:58.412483 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412460 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2xdm\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412494 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412513 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412552 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412610 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412605 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412641 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412696 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412727 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.412864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.412764 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.413260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.413227 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.413260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.413252 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.413455 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.413402 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.414324 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.414297 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.414432 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.414405 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.415532 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.415508 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.415846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.415823 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.473555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.473515 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2xdm\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.473769 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.473673 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\") pod \"e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:58.615762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:58.615544 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:52:59.040398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:59.040375 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 21:52:59.041835 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:52:59.041811 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803259bc_9cdf_4d90_affb_b16e1b3b8f0f.slice/crio-9c29a6ea138df712baeaadb4b46c0b849009ae00a6a93e3a3ed82f61e3fa07b0 WatchSource:0}: Error finding container 9c29a6ea138df712baeaadb4b46c0b849009ae00a6a93e3a3ed82f61e3fa07b0: Status 404 returned error can't find the container with id 9c29a6ea138df712baeaadb4b46c0b849009ae00a6a93e3a3ed82f61e3fa07b0 Nov 02 21:52:59.044186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:59.044153 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:59.044259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:59.044218 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:59.044259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:52:59.044247 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:00.013365 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:00.013330 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerStarted","Data":"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0"} Nov 02 21:53:00.013740 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:00.013391 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerStarted","Data":"9c29a6ea138df712baeaadb4b46c0b849009ae00a6a93e3a3ed82f61e3fa07b0"} Nov 02 21:53:00.232323 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:00.232241 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podStartSLOduration=2.23222834 podStartE2EDuration="2.23222834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:00.181302197 +0000 UTC m=+2619.729551997" watchObservedRunningTime="2026-11-02 21:53:00.23222834 +0000 UTC m=+2619.780478180" Nov 02 21:53:00.616291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:00.616265 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:53:00.621766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:00.621743 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:53:01.022043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.021938 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:53:01.022960 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.022928 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:53:01.572523 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.572489 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:53:01.576833 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.576817 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650684 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650654 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qstdw\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650775 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650835 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650801 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650874 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650896 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.650980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.650968 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.651147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.651029 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.651147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.651048 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.651147 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.651074 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.671364 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.671337 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:53:01.752195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752130 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752195 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752183 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qstdw\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752262 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752284 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752337 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752367 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752414 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752510 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752540 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752709 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752768 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752736 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752804 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.752992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.752937 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.753173 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.753149 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.754658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.754636 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.754849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.754831 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.810202 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.810179 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.810484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.810463 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qstdw\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:01.891167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:01.891112 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:02.326568 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:02.326545 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:53:02.327679 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:53:02.327657 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5bace8_1b13_476c_8056_ca50b3150a4b.slice/crio-7f39cec1fc41f44f34a22a8ab81b8f65a9a61b8ee28b1e9155235b43ce31112a WatchSource:0}: Error finding container 7f39cec1fc41f44f34a22a8ab81b8f65a9a61b8ee28b1e9155235b43ce31112a: Status 404 returned error can't find the container with id 7f39cec1fc41f44f34a22a8ab81b8f65a9a61b8ee28b1e9155235b43ce31112a Nov 02 21:53:02.329939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:02.329905 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.330035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:02.329996 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.330035 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:02.330029 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:03.035844 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.035811 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" event={"ID":"0c5bace8-1b13-476c-8056-ca50b3150a4b","Type":"ContainerStarted","Data":"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7"} Nov 02 21:53:03.036049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.035867 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" event={"ID":"0c5bace8-1b13-476c-8056-ca50b3150a4b","Type":"ContainerStarted","Data":"7f39cec1fc41f44f34a22a8ab81b8f65a9a61b8ee28b1e9155235b43ce31112a"} Nov 02 21:53:03.157451 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.157399 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podStartSLOduration=2.157385413 podStartE2EDuration="2.157385413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.15566571 +0000 UTC m=+2622.703915508" watchObservedRunningTime="2026-11-02 21:53:03.157385413 +0000 UTC m=+2622.705635193" Nov 02 21:53:03.891918 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.891880 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:03.893667 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.893641 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:03.893792 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:03.893699 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:04.891629 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:04.891602 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:04.891783 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:04.891647 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:05.891901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:05.891864 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:05.892310 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:05.891922 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:06.892062 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:06.892027 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:06.892644 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:06.892606 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:07.131107 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.131070 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 21:53:07.136046 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.136024 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.173129 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.173103 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 21:53:07.241205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.241172 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6e535d\"" Nov 02 21:53:07.250604 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.249162 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82p8\" (UniqueName: \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.250972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.250935 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.251277 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.250354 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6e535d-dockercfg-5lm47\"" Nov 02 21:53:07.352421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.352386 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82p8\" (UniqueName: \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.352736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.352712 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.353403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.353384 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.432527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.432418 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82p8\" (UniqueName: \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\") pod \"payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.453167 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.453141 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:07.850534 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.850509 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 21:53:07.851471 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:53:07.851437 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e5796c_0890_40f2_91b0_19f8669d0524.slice/crio-c23bfd3bc80a4c7310253de263a025c71b9d417347fb03d0663def306b66e08c WatchSource:0}: Error finding container c23bfd3bc80a4c7310253de263a025c71b9d417347fb03d0663def306b66e08c: Status 404 returned error can't find the container with id c23bfd3bc80a4c7310253de263a025c71b9d417347fb03d0663def306b66e08c Nov 02 21:53:07.891821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.891798 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.891901 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:07.891839 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:08.002676 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.002653 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 21:53:08.078492 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.078452 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" event={"ID":"82e5796c-0890-40f2-91b0-19f8669d0524","Type":"ContainerStarted","Data":"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4"} Nov 02 21:53:08.078657 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.078561 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" event={"ID":"82e5796c-0890-40f2-91b0-19f8669d0524","Type":"ContainerStarted","Data":"c23bfd3bc80a4c7310253de263a025c71b9d417347fb03d0663def306b66e08c"} Nov 02 21:53:08.078840 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.078822 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:08.236739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.236637 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" podStartSLOduration=1.236622446 podStartE2EDuration="1.236622446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:08.234484709 +0000 UTC m=+2627.782734490" watchObservedRunningTime="2026-11-02 21:53:08.236622446 +0000 UTC m=+2627.784872268" Nov 02 21:53:08.892108 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.892071 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 02 21:53:08.892311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:08.892141 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 02 21:53:09.987817 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:09.987787 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:09.988488 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:09.988258 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:09.989057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:09.989038 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:53:12.017680 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.017649 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 21:53:12.023042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.023008 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.058268 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.058227 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 21:53:12.111354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.111313 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-7cde84-dockercfg-9qwlf\"" Nov 02 21:53:12.115313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.111576 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-7cde84\"" Nov 02 21:53:12.115997 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.115933 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k649b\" (UniqueName: \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.116190 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.116171 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.217221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.217181 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k649b\" (UniqueName: \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.218197 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.218168 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.219461 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.219085 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.274417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.274334 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k649b\" (UniqueName: \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\") pod \"payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.339503 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.339471 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:12.732010 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:12.722032 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 21:53:12.732010 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:53:12.723179 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fb04b2_2384_4c92_ac02_15a6c5c67eab.slice/crio-ac0d196da2203ae17dd883c4e72db57e97ce36f7b14f0eb7fb61a2767f34bd7b WatchSource:0}: Error finding container ac0d196da2203ae17dd883c4e72db57e97ce36f7b14f0eb7fb61a2767f34bd7b: Status 404 returned error can't find the container with id ac0d196da2203ae17dd883c4e72db57e97ce36f7b14f0eb7fb61a2767f34bd7b Nov 02 21:53:13.178614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:13.178576 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" event={"ID":"58fb04b2-2384-4c92-ac02-15a6c5c67eab","Type":"ContainerStarted","Data":"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e"} Nov 02 21:53:13.183075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:13.183046 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" event={"ID":"58fb04b2-2384-4c92-ac02-15a6c5c67eab","Type":"ContainerStarted","Data":"ac0d196da2203ae17dd883c4e72db57e97ce36f7b14f0eb7fb61a2767f34bd7b"} Nov 02 21:53:13.186288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:13.185927 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:13.307691 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:13.307571 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" podStartSLOduration=2.307552959 podStartE2EDuration="2.307552959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:13.305288835 +0000 UTC m=+2632.853538616" watchObservedRunningTime="2026-11-02 21:53:13.307552959 +0000 UTC m=+2632.855802740" Nov 02 21:53:14.782973 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:14.782017 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n" podUID="8e0ac144-950e-424b-9461-6a2333d05969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.796892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:14.795432 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:53:19.089669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:19.089642 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 21:53:25.230970 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:25.230916 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 21:53:25.944940 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:53:25.944906 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:53:33.351024 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.350990 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 21:53:33.356834 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.356810 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398547 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398521 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398572 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398692 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398665 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398718 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398745 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.398821 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398811 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.399042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.398857 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwgz\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.409536 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.409508 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:53:33.409650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.409508 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:53:33.409650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.409604 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n4jrm\"" Nov 02 21:53:33.409650 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.409619 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:53:33.464829 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.464800 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 21:53:33.499464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499432 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499485 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499530 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499570 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499557 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499688 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499613 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499683 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwgz\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.499818 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.499792 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.500291 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.500264 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.501913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.501884 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.501913 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.501895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.502286 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.502267 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.502380 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.502315 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.556499 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.556476 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwgz\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.557023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.557004 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c882q\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:33.672453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:33.672404 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:34.100520 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.100488 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 21:53:34.138713 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:53:34.138689 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d2f6d_011f_4056_a9a4_41849d07e76c.slice/crio-35f8626dddc1b45de47b0158284b04194eec014a52dcc538113388912f5b65c8 WatchSource:0}: Error finding container 35f8626dddc1b45de47b0158284b04194eec014a52dcc538113388912f5b65c8: Status 404 returned error can't find the container with id 35f8626dddc1b45de47b0158284b04194eec014a52dcc538113388912f5b65c8 Nov 02 21:53:34.141328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.141297 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:34.141410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.141356 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:34.350800 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.350724 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" event={"ID":"367d2f6d-011f-4056-a9a4-41849d07e76c","Type":"ContainerStarted","Data":"60e2639cceded820b180d8c76b0cbe0fa68eb10b5491307b9051eb0dfd268c73"} Nov 02 21:53:34.350800 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.350768 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" event={"ID":"367d2f6d-011f-4056-a9a4-41849d07e76c","Type":"ContainerStarted","Data":"35f8626dddc1b45de47b0158284b04194eec014a52dcc538113388912f5b65c8"} Nov 02 21:53:34.351009 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.350892 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:34.469778 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:34.469731 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" podStartSLOduration=1.469717447 podStartE2EDuration="1.469717447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:34.466918974 +0000 UTC m=+2654.015168779" watchObservedRunningTime="2026-11-02 21:53:34.469717447 +0000 UTC m=+2654.017967226" Nov 02 21:53:35.358970 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:35.358928 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c882q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:53:35.359115 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:35.359007 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:53:38.358599 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:38.358563 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 21:53:51.370690 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:51.370657 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:53:55.946575 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:53:55.946538 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:53:56.371870 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:53:56.371838 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" containerID="cri-o://a46b2644b70b598c1e84aabad603d5b0859505ff2b23f05e4e3744948bceff56" gracePeriod=25 Nov 02 21:54:03.046719 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:03.046678 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:54:07.400266 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.400226 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:54:07.400849 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.400445 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" podUID="945da409-b927-4d49-94cf-83a4655f5b44" containerName="maas-api" containerID="cri-o://f8d889564238077a58ffdafd4a55e37fd13473e5b7843b71f66d1768da856dbf" gracePeriod=30 Nov 02 21:54:07.590605 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.590574 5859 generic.go:356] "Generic (PLEG): container finished" podID="945da409-b927-4d49-94cf-83a4655f5b44" containerID="f8d889564238077a58ffdafd4a55e37fd13473e5b7843b71f66d1768da856dbf" exitCode=0 Nov 02 21:54:07.590762 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.590652 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" event={"ID":"945da409-b927-4d49-94cf-83a4655f5b44","Type":"ContainerDied","Data":"f8d889564238077a58ffdafd4a55e37fd13473e5b7843b71f66d1768da856dbf"} Nov 02 21:54:07.680596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.680572 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.811903 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\") pod \"945da409-b927-4d49-94cf-83a4655f5b44\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.811984 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\") pod \"945da409-b927-4d49-94cf-83a4655f5b44\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.812042 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") pod \"945da409-b927-4d49-94cf-83a4655f5b44\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.812219 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\" (UniqueName: \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\") pod \"945da409-b927-4d49-94cf-83a4655f5b44\" (UID: \"945da409-b927-4d49-94cf-83a4655f5b44\") " Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.812538 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca" pod "945da409-b927-4d49-94cf-83a4655f5b44" (UID: "945da409-b927-4d49-94cf-83a4655f5b44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:07.813186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.812857 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/945da409-b927-4d49-94cf-83a4655f5b44-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:07.816246 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.816204 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr" pod "945da409-b927-4d49-94cf-83a4655f5b44" (UID: "945da409-b927-4d49-94cf-83a4655f5b44"). InnerVolumeSpecName "kube-api-access-2nbtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:07.822906 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.822863 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs" pod "945da409-b927-4d49-94cf-83a4655f5b44" (UID: "945da409-b927-4d49-94cf-83a4655f5b44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:07.827434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.827407 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls" pod "945da409-b927-4d49-94cf-83a4655f5b44" (UID: "945da409-b927-4d49-94cf-83a4655f5b44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:07.914625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.914550 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:07.914625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.914580 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/945da409-b927-4d49-94cf-83a4655f5b44-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:07.914625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:07.914597 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbtr\" (UniqueName: \"kubernetes.io/projected/945da409-b927-4d49-94cf-83a4655f5b44-kube-api-access-2nbtr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:08.598572 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:08.598542 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" Nov 02 21:54:08.598572 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:08.598558 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb" event={"ID":"945da409-b927-4d49-94cf-83a4655f5b44","Type":"ContainerDied","Data":"d57b54ea78792a565017ff7e18dca9a3715bd8f3a9c76ced742137b27a34639b"} Nov 02 21:54:08.599043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:08.598612 5859 scope.go:122] "RemoveContainer" containerID="f8d889564238077a58ffdafd4a55e37fd13473e5b7843b71f66d1768da856dbf" Nov 02 21:54:08.764419 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:08.764384 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:54:08.814685 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:08.814655 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eabda2-77ff5784d7-ljshb"] Nov 02 21:54:10.571505 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:10.571473 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945da409-b927-4d49-94cf-83a4655f5b44" path="/var/lib/kubelet/pods/945da409-b927-4d49-94cf-83a4655f5b44/volumes" Nov 02 21:54:13.046782 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:13.046739 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:54:16.650823 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.650793 5859 generic.go:356] "Generic (PLEG): container finished" podID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerID="a46b2644b70b598c1e84aabad603d5b0859505ff2b23f05e4e3744948bceff56" exitCode=0 Nov 02 21:54:16.651201 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.650861 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" event={"ID":"3cf9ba23-e654-412e-ba41-e96a4e054da7","Type":"ContainerDied","Data":"a46b2644b70b598c1e84aabad603d5b0859505ff2b23f05e4e3744948bceff56"} Nov 02 21:54:16.674851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.674832 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:54:16.843332 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.843306 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\") pod \"3cf9ba23-e654-412e-ba41-e96a4e054da7\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " Nov 02 21:54:16.843491 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.843400 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\" (UniqueName: \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\") pod \"3cf9ba23-e654-412e-ba41-e96a4e054da7\" (UID: \"3cf9ba23-e654-412e-ba41-e96a4e054da7\") " Nov 02 21:54:16.843651 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.843627 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume" pod "3cf9ba23-e654-412e-ba41-e96a4e054da7" (UID: "3cf9ba23-e654-412e-ba41-e96a4e054da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:16.843857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.843843 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf9ba23-e654-412e-ba41-e96a4e054da7-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:16.845428 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.845405 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx" pod "3cf9ba23-e654-412e-ba41-e96a4e054da7" (UID: "3cf9ba23-e654-412e-ba41-e96a4e054da7"). InnerVolumeSpecName "kube-api-access-rxpqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:16.945522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:16.945465 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpqx\" (UniqueName: \"kubernetes.io/projected/3cf9ba23-e654-412e-ba41-e96a4e054da7-kube-api-access-rxpqx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:17.658696 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:17.658669 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" Nov 02 21:54:17.659151 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:17.658712 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl" event={"ID":"3cf9ba23-e654-412e-ba41-e96a4e054da7","Type":"ContainerDied","Data":"3975863ea770db9971a87aeadf8820f6060dad219e36bfcc80c1be9291b19bc8"} Nov 02 21:54:17.659151 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:17.658767 5859 scope.go:122] "RemoveContainer" containerID="a46b2644b70b598c1e84aabad603d5b0859505ff2b23f05e4e3744948bceff56" Nov 02 21:54:17.823098 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:17.823072 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:54:17.873162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:17.873139 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-aa8aff-7c48f54ccf-wswnl"] Nov 02 21:54:18.570933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:18.570896 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" path="/var/lib/kubelet/pods/3cf9ba23-e654-412e-ba41-e96a4e054da7/volumes" Nov 02 21:54:25.063048 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.063015 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:54:25.063522 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.063205 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" podUID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" containerName="maas-api" containerID="cri-o://1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035" gracePeriod=30 Nov 02 21:54:25.328566 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.328529 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:54:25.432848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.432810 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\" (UniqueName: \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\") pod \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " Nov 02 21:54:25.433049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.432889 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\") pod \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " Nov 02 21:54:25.433049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.432923 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\") pod \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " Nov 02 21:54:25.433237 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.433112 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\") pod \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\" (UID: \"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0\") " Nov 02 21:54:25.433698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.433658 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca" pod "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" (UID: "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:25.433853 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.433835 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:25.435532 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.435508 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls" pod "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" (UID: "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:25.436081 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.436048 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb" pod "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" (UID: "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0"). InnerVolumeSpecName "kube-api-access-z69wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:25.436311 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.436289 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs" pod "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" (UID: "93c4e8e2-94ad-49eb-b9ef-5821cc905aa0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:25.534772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.534746 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:25.534772 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.534772 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z69wb\" (UniqueName: \"kubernetes.io/projected/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-kube-api-access-z69wb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:25.534931 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.534784 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:25.712139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.712054 5859 generic.go:356] "Generic (PLEG): container finished" podID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" containerID="1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035" exitCode=0 Nov 02 21:54:25.712139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.712121 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" event={"ID":"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0","Type":"ContainerDied","Data":"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035"} Nov 02 21:54:25.712334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.712112 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" Nov 02 21:54:25.712334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.712169 5859 scope.go:122] "RemoveContainer" containerID="1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035" Nov 02 21:54:25.712334 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.712159 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw" event={"ID":"93c4e8e2-94ad-49eb-b9ef-5821cc905aa0","Type":"ContainerDied","Data":"6e136827a4517deb6ac04254bec9627af84714916216d3494c97691adcd98737"} Nov 02 21:54:25.726343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.726320 5859 scope.go:122] "RemoveContainer" containerID="1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035" Nov 02 21:54:25.726656 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:25.726633 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035\": container with ID starting with 1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035 not found: ID does not exist" containerID="1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035" Nov 02 21:54:25.726769 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.726664 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035"} err="failed to get container status \"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035\": rpc error: code = NotFound desc = could not find container \"1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035\": container with ID starting with 1cb41e1027d572fdb704b1a1062967dbc0b8c912ca3402f4dcd6414f4ad77035 not found: ID does not exist" Nov 02 21:54:25.878083 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.878058 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:54:25.904751 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:25.904726 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3c7706-86ccf9c89-zxhqw"] Nov 02 21:54:25.951489 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:25.951454 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:54:26.570622 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:26.570587 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" path="/var/lib/kubelet/pods/93c4e8e2-94ad-49eb-b9ef-5821cc905aa0/volumes" Nov 02 21:54:39.808038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.808010 5859 generic.go:356] "Generic (PLEG): container finished" podID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerID="9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0" exitCode=0 Nov 02 21:54:39.808387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.808090 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerDied","Data":"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0"} Nov 02 21:54:39.808476 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.808459 5859 scope.go:122] "RemoveContainer" containerID="9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0" Nov 02 21:54:39.809794 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.809768 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:39.809864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.809834 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:39.809899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:39.809887 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:40.835795 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:40.835758 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerStarted","Data":"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea"} Nov 02 21:54:41.620984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:41.620340 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:54:41.626399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:41.626369 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 21:54:41.626608 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:41.626577 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 21:54:41.957448 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:41.957324 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:54:41.957937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:41.957584 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" podUID="8b03d709-7126-4207-85a4-92a861f77f3c" containerName="maas-api" containerID="cri-o://bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430" gracePeriod=30 Nov 02 21:54:42.280359 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.280333 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:54:42.359012 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.358943 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") pod \"8b03d709-7126-4207-85a4-92a861f77f3c\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " Nov 02 21:54:42.359207 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.359065 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\") pod \"8b03d709-7126-4207-85a4-92a861f77f3c\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " Nov 02 21:54:42.359207 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.359139 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\" (UniqueName: \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\") pod \"8b03d709-7126-4207-85a4-92a861f77f3c\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " Nov 02 21:54:42.359329 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.359217 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\") pod \"8b03d709-7126-4207-85a4-92a861f77f3c\" (UID: \"8b03d709-7126-4207-85a4-92a861f77f3c\") " Nov 02 21:54:42.359524 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.359488 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca" pod "8b03d709-7126-4207-85a4-92a861f77f3c" (UID: "8b03d709-7126-4207-85a4-92a861f77f3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:42.359884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.359858 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b03d709-7126-4207-85a4-92a861f77f3c-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:42.361943 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.361914 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs" pod "8b03d709-7126-4207-85a4-92a861f77f3c" (UID: "8b03d709-7126-4207-85a4-92a861f77f3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:42.362217 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.362187 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h" pod "8b03d709-7126-4207-85a4-92a861f77f3c" (UID: "8b03d709-7126-4207-85a4-92a861f77f3c"). InnerVolumeSpecName "kube-api-access-pg45h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:42.362434 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.362414 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls" pod "8b03d709-7126-4207-85a4-92a861f77f3c" (UID: "8b03d709-7126-4207-85a4-92a861f77f3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:42.461193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.461153 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:42.461193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.461190 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg45h\" (UniqueName: \"kubernetes.io/projected/8b03d709-7126-4207-85a4-92a861f77f3c-kube-api-access-pg45h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:42.461193 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.461202 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b03d709-7126-4207-85a4-92a861f77f3c-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:42.627972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.627346 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 21:54:42.627972 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.627417 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 21:54:42.850435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.850352 5859 generic.go:356] "Generic (PLEG): container finished" podID="8b03d709-7126-4207-85a4-92a861f77f3c" containerID="bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430" exitCode=0 Nov 02 21:54:42.850435 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.850407 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" Nov 02 21:54:42.850625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.850431 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" event={"ID":"8b03d709-7126-4207-85a4-92a861f77f3c","Type":"ContainerDied","Data":"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430"} Nov 02 21:54:42.850625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.850495 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z" event={"ID":"8b03d709-7126-4207-85a4-92a861f77f3c","Type":"ContainerDied","Data":"efa7342d34deaa674a562c56e225a0d0c9e74c125d0b56a4397de562a743c729"} Nov 02 21:54:42.850625 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.850521 5859 scope.go:122] "RemoveContainer" containerID="bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430" Nov 02 21:54:42.860654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.860636 5859 scope.go:122] "RemoveContainer" containerID="bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430" Nov 02 21:54:42.860935 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:42.860913 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430\": container with ID starting with bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430 not found: ID does not exist" containerID="bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430" Nov 02 21:54:42.861027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:42.860944 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430"} err="failed to get container status \"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430\": rpc error: code = NotFound desc = could not find container \"bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430\": container with ID starting with bd11ad9946153063b1f9cc217686d5804cd52d729d7baae9e2eda35a0a115430 not found: ID does not exist" Nov 02 21:54:43.047074 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:43.047032 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:54:43.047642 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:43.047621 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-aa8aff-55bcd8d7cc-k5x5z"] Nov 02 21:54:43.625360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:43.625325 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 21:54:43.625639 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:43.625609 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 21:54:44.571638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:44.571602 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b03d709-7126-4207-85a4-92a861f77f3c" path="/var/lib/kubelet/pods/8b03d709-7126-4207-85a4-92a861f77f3c/volumes" Nov 02 21:54:44.623327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:44.623296 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 21:54:44.623611 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:44.623584 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 21:54:45.616298 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:45.616263 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 21:54:45.616674 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:45.616315 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 21:54:46.021941 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:46.021857 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:54:46.620376 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:46.620348 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:54:46.878715 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:46.878635 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 21:54:46.992538 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:46.992503 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:54:46.992774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:46.992751 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" containerID="cri-o://aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c" gracePeriod=30 Nov 02 21:54:52.258257 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.258198 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:54:52.397645 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397615 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.397774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397702 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.397774 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397731 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.397876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397771 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.397876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397810 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.397876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397847 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.398075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397873 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.398075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397911 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.398075 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398031 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\") pod \"e8687284-6702-4573-a8d6-c5b5464ab9f6\" (UID: \"e8687284-6702-4573-a8d6-c5b5464ab9f6\") " Nov 02 21:54:52.398233 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398186 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:52.398293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398251 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:52.398487 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398463 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:52.398592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.397865 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:52.398592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398576 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:52.398718 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398697 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.398790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398724 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.398790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398738 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8687284-6702-4573-a8d6-c5b5464ab9f6-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.398790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.398751 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.400669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.400619 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:54:52.400669 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.400624 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:54:52.400892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.400873 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "kube-api-access-bd9v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:52.400980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.400910 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token" pod "e8687284-6702-4573-a8d6-c5b5464ab9f6" (UID: "e8687284-6702-4573-a8d6-c5b5464ab9f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:52.500068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.500043 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.500068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.500066 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.500224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.500077 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.500224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.500094 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8687284-6702-4573-a8d6-c5b5464ab9f6-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.500224 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.500103 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9v9\" (UniqueName: \"kubernetes.io/projected/e8687284-6702-4573-a8d6-c5b5464ab9f6-kube-api-access-bd9v9\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:52.919721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.919647 5859 generic.go:356] "Generic (PLEG): container finished" podID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerID="aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c" exitCode=0 Nov 02 21:54:52.919721 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.919699 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" Nov 02 21:54:52.919905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.919730 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" event={"ID":"e8687284-6702-4573-a8d6-c5b5464ab9f6","Type":"ContainerDied","Data":"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c"} Nov 02 21:54:52.919905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.919804 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75" event={"ID":"e8687284-6702-4573-a8d6-c5b5464ab9f6","Type":"ContainerDied","Data":"7a575e78a990c474ac312a62d28884a88f911fc4ff5bcd8b26e75a5d9e04db72"} Nov 02 21:54:52.919905 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.919829 5859 scope.go:122] "RemoveContainer" containerID="aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c" Nov 02 21:54:52.931256 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.931239 5859 scope.go:122] "RemoveContainer" containerID="aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c" Nov 02 21:54:52.931522 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:52.931505 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c\": container with ID starting with aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c not found: ID does not exist" containerID="aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c" Nov 02 21:54:52.931565 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:52.931529 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c"} err="failed to get container status \"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c\": rpc error: code = NotFound desc = could not find container \"aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c\": container with ID starting with aa71fe9501c48e7c5ced203ef75eb3ba90a6512b670a344a2024d33ee8b95a1c not found: ID does not exist" Nov 02 21:54:53.131007 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:53.130975 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:54:53.181403 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:53.181347 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-aa8aff-openshift-default-5dd5b7d6df-67x75"] Nov 02 21:54:54.009184 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.009153 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:54:54.009561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.009364 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" podUID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" containerName="maas-api" containerID="cri-o://8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb" gracePeriod=30 Nov 02 21:54:54.336199 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.336167 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:54:54.345355 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.345325 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:54:54.434769 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.434743 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\") pod \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " Nov 02 21:54:54.434937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.434806 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\" (UniqueName: \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\") pod \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " Nov 02 21:54:54.434937 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.434931 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") pod \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " Nov 02 21:54:54.435126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.434985 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\") pod \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\" (UID: \"6eef6c43-6199-4e5c-9f26-5ccac44e59d5\") " Nov 02 21:54:54.435183 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.435148 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca" pod "6eef6c43-6199-4e5c-9f26-5ccac44e59d5" (UID: "6eef6c43-6199-4e5c-9f26-5ccac44e59d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:54.435565 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.435535 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:54.437042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.437017 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5" pod "6eef6c43-6199-4e5c-9f26-5ccac44e59d5" (UID: "6eef6c43-6199-4e5c-9f26-5ccac44e59d5"). InnerVolumeSpecName "kube-api-access-clzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:54.437042 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.437025 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls" pod "6eef6c43-6199-4e5c-9f26-5ccac44e59d5" (UID: "6eef6c43-6199-4e5c-9f26-5ccac44e59d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:54.437206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.437189 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs" pod "6eef6c43-6199-4e5c-9f26-5ccac44e59d5" (UID: "6eef6c43-6199-4e5c-9f26-5ccac44e59d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:54:54.451852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.451826 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:54:54.536531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.536462 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzs5\" (UniqueName: \"kubernetes.io/projected/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-kube-api-access-clzs5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:54.536531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.536489 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:54.536531 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.536500 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eef6c43-6199-4e5c-9f26-5ccac44e59d5-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:54.571351 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.571323 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" path="/var/lib/kubelet/pods/e8687284-6702-4573-a8d6-c5b5464ab9f6/volumes" Nov 02 21:54:54.936485 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.936436 5859 generic.go:356] "Generic (PLEG): container finished" podID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" containerID="8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb" exitCode=0 Nov 02 21:54:54.936739 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.936507 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" Nov 02 21:54:54.936859 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.936521 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" event={"ID":"6eef6c43-6199-4e5c-9f26-5ccac44e59d5","Type":"ContainerDied","Data":"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb"} Nov 02 21:54:54.937014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.936993 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8" event={"ID":"6eef6c43-6199-4e5c-9f26-5ccac44e59d5","Type":"ContainerDied","Data":"5ba130287c1e2bacbaef6a0d6dec2f7e579e7ad45c3698f29e4b528d683d8984"} Nov 02 21:54:54.937136 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.937036 5859 scope.go:122] "RemoveContainer" containerID="8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb" Nov 02 21:54:54.948105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.948074 5859 scope.go:122] "RemoveContainer" containerID="8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb" Nov 02 21:54:54.948423 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:54.948403 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb\": container with ID starting with 8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb not found: ID does not exist" containerID="8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb" Nov 02 21:54:54.948521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:54.948430 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb"} err="failed to get container status \"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb\": rpc error: code = NotFound desc = could not find container \"8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb\": container with ID starting with 8fed3b04d5a2a06acea8e7945af0d8ab0d1c6385318ffd7856b4adae07ad66cb not found: ID does not exist" Nov 02 21:54:55.104225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:55.104192 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:54:55.113869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:55.113839 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb45bc-548cb994c5-ct5h8"] Nov 02 21:54:55.947651 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:55.947619 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:54:56.579409 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:56.579376 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" path="/var/lib/kubelet/pods/6eef6c43-6199-4e5c-9f26-5ccac44e59d5/volumes" Nov 02 21:54:59.346235 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.346194 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" podUID="ee42edad-5982-436e-a25f-edd440443331" containerName="payload-processing" containerID="cri-o://16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94" gracePeriod=25 Nov 02 21:54:59.452729 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.452695 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" podUID="4c436fcd-a610-4835-b495-016edb69349c" containerName="payload-pre-processing" containerID="cri-o://49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc" gracePeriod=25 Nov 02 21:54:59.603890 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.603826 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:54:59.709861 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.709835 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\") pod \"ee42edad-5982-436e-a25f-edd440443331\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " Nov 02 21:54:59.710105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.710087 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\" (UniqueName: \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\") pod \"ee42edad-5982-436e-a25f-edd440443331\" (UID: \"ee42edad-5982-436e-a25f-edd440443331\") " Nov 02 21:54:59.710288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.710260 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume" pod "ee42edad-5982-436e-a25f-edd440443331" (UID: "ee42edad-5982-436e-a25f-edd440443331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:59.710591 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.710575 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee42edad-5982-436e-a25f-edd440443331-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:59.712415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.712394 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns" pod "ee42edad-5982-436e-a25f-edd440443331" (UID: "ee42edad-5982-436e-a25f-edd440443331"). InnerVolumeSpecName "kube-api-access-pfcns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:59.722922 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.722886 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:54:59.811592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.811568 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\") pod \"4c436fcd-a610-4835-b495-016edb69349c\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " Nov 02 21:54:59.811730 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.811607 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\" (UniqueName: \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\") pod \"4c436fcd-a610-4835-b495-016edb69349c\" (UID: \"4c436fcd-a610-4835-b495-016edb69349c\") " Nov 02 21:54:59.811942 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.811920 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume" pod "4c436fcd-a610-4835-b495-016edb69349c" (UID: "4c436fcd-a610-4835-b495-016edb69349c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:59.812162 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.812146 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c436fcd-a610-4835-b495-016edb69349c-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:59.812214 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.812167 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfcns\" (UniqueName: \"kubernetes.io/projected/ee42edad-5982-436e-a25f-edd440443331-kube-api-access-pfcns\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:59.813596 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.813577 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b" pod "4c436fcd-a610-4835-b495-016edb69349c" (UID: "4c436fcd-a610-4835-b495-016edb69349c"). InnerVolumeSpecName "kube-api-access-h9l9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:59.913108 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.913084 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9l9b\" (UniqueName: \"kubernetes.io/projected/4c436fcd-a610-4835-b495-016edb69349c-kube-api-access-h9l9b\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:59.974532 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.974509 5859 generic.go:356] "Generic (PLEG): container finished" podID="4c436fcd-a610-4835-b495-016edb69349c" containerID="49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc" exitCode=0 Nov 02 21:54:59.974646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.974554 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" Nov 02 21:54:59.974646 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.974592 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" event={"ID":"4c436fcd-a610-4835-b495-016edb69349c","Type":"ContainerDied","Data":"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc"} Nov 02 21:54:59.974731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.974645 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k" event={"ID":"4c436fcd-a610-4835-b495-016edb69349c","Type":"ContainerDied","Data":"1ce0944e8fbf2bc129b6ae32807ec4b234cffeb4484793af8dc070240443cff5"} Nov 02 21:54:59.974731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.974675 5859 scope.go:122] "RemoveContainer" containerID="49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc" Nov 02 21:54:59.976133 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.976110 5859 generic.go:356] "Generic (PLEG): container finished" podID="ee42edad-5982-436e-a25f-edd440443331" containerID="16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94" exitCode=0 Nov 02 21:54:59.976242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.976153 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" Nov 02 21:54:59.976242 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.976153 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" event={"ID":"ee42edad-5982-436e-a25f-edd440443331","Type":"ContainerDied","Data":"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94"} Nov 02 21:54:59.976366 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.976275 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9" event={"ID":"ee42edad-5982-436e-a25f-edd440443331","Type":"ContainerDied","Data":"29cdaa4819005d4fc3fd24612fb6ca50ef9b34a7d991a18e5d5b42f89fdbae26"} Nov 02 21:54:59.986256 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.986237 5859 scope.go:122] "RemoveContainer" containerID="49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc" Nov 02 21:54:59.986537 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:59.986512 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc\": container with ID starting with 49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc not found: ID does not exist" containerID="49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc" Nov 02 21:54:59.986630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.986543 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc"} err="failed to get container status \"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc\": rpc error: code = NotFound desc = could not find container \"49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc\": container with ID starting with 49d04883ce35dd72410ed5faf241a9744264f112f7727f49f9f0abffbb8994dc not found: ID does not exist" Nov 02 21:54:59.986630 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.986560 5859 scope.go:122] "RemoveContainer" containerID="16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94" Nov 02 21:54:59.995487 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.995469 5859 scope.go:122] "RemoveContainer" containerID="16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94" Nov 02 21:54:59.995753 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:54:59.995732 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94\": container with ID starting with 16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94 not found: ID does not exist" containerID="16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94" Nov 02 21:54:59.995860 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:54:59.995757 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94"} err="failed to get container status \"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94\": rpc error: code = NotFound desc = could not find container \"16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94\": container with ID starting with 16f7d3ac7ce8081c8f377ed705a9394d5166de562b62efbf1b60e0988c1a4e94 not found: ID does not exist" Nov 02 21:55:00.138806 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.138730 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:55:00.141583 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.141563 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb45bc-84594bd6c5-5cc5k"] Nov 02 21:55:00.260899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.260872 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:55:00.275529 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.275505 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bb45bc-79b545db97-2bbv9"] Nov 02 21:55:00.572217 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.572184 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c436fcd-a610-4835-b495-016edb69349c" path="/var/lib/kubelet/pods/4c436fcd-a610-4835-b495-016edb69349c/volumes" Nov 02 21:55:00.572748 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.572727 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee42edad-5982-436e-a25f-edd440443331" path="/var/lib/kubelet/pods/ee42edad-5982-436e-a25f-edd440443331/volumes" Nov 02 21:55:00.968571 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.968488 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:55:00.968816 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:00.968785 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" containerID="cri-o://f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc" gracePeriod=30 Nov 02 21:55:01.086282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:01.086192 5859 patch_prober.go:29] interesting pod/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:55:01.086282 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:01.086247 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:55:06.287573 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.287541 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:55:06.406884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.406847 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.406884 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.406889 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.406964 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407109 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407091 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407152 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407181 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407236 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407245 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:06.407318 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407278 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407345 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\") pod \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\" (UID: \"08c57a72-83b4-4ac8-9ed7-17ff0eba540f\") " Nov 02 21:55:06.407477 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407461 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:06.407580 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407512 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:06.407699 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407617 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:06.407764 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.407731 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:06.408041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.408016 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.408041 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.408041 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.408247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.408055 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.408247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.408070 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.408247 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.408089 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.409353 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.409328 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:55:06.409466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.409334 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:06.409682 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.409664 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "kube-api-access-xgc6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:06.409731 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.409702 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token" pod "08c57a72-83b4-4ac8-9ed7-17ff0eba540f" (UID: "08c57a72-83b4-4ac8-9ed7-17ff0eba540f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:06.509820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.509737 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.509820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.509767 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgc6b\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-kube-api-access-xgc6b\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.509820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.509777 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:06.509820 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:06.509786 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c57a72-83b4-4ac8-9ed7-17ff0eba540f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:07.032673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.032640 5859 generic.go:356] "Generic (PLEG): container finished" podID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerID="f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc" exitCode=0 Nov 02 21:55:07.032872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.032681 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" Nov 02 21:55:07.032872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.032726 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerDied","Data":"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc"} Nov 02 21:55:07.032872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.032785 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb" event={"ID":"08c57a72-83b4-4ac8-9ed7-17ff0eba540f","Type":"ContainerDied","Data":"ccd8df8a8bbb507c56d0f365921de8c6540008c6a1423f83a1abf39a744975ea"} Nov 02 21:55:07.032872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.032814 5859 scope.go:122] "RemoveContainer" containerID="f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc" Nov 02 21:55:07.045338 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.045313 5859 scope.go:122] "RemoveContainer" containerID="5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec" Nov 02 21:55:07.054744 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.054724 5859 scope.go:122] "RemoveContainer" containerID="f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc" Nov 02 21:55:07.055047 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:07.055017 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc\": container with ID starting with f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc not found: ID does not exist" containerID="f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc" Nov 02 21:55:07.055137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.055056 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc"} err="failed to get container status \"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc\": rpc error: code = NotFound desc = could not find container \"f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc\": container with ID starting with f51ca21b8662c57e3ac715a684e88f79198f02d1231bbff5c3777ae4c06799fc not found: ID does not exist" Nov 02 21:55:07.055137 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.055081 5859 scope.go:122] "RemoveContainer" containerID="5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec" Nov 02 21:55:07.055367 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:07.055349 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec\": container with ID starting with 5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec not found: ID does not exist" containerID="5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec" Nov 02 21:55:07.055417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.055373 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec"} err="failed to get container status \"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec\": rpc error: code = NotFound desc = could not find container \"5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec\": container with ID starting with 5be22deb4d327e24002fc77efc8a42d20bf904d5f8ff8b898cafde97ec0636ec not found: ID does not exist" Nov 02 21:55:07.227416 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.227380 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:55:07.230025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:07.229998 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb45bc-openshift-default-cc8dbbcb9-6m6jb"] Nov 02 21:55:08.570619 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:08.570585 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" path="/var/lib/kubelet/pods/08c57a72-83b4-4ac8-9ed7-17ff0eba540f/volumes" Nov 02 21:55:13.665188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665156 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 21:55:13.665678 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665665 5859 cpu_manager.go:414] "containerMap: removing container" podUID="8b03d709-7126-4207-85a4-92a861f77f3c" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665679 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b03d709-7126-4207-85a4-92a861f77f3c" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665690 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665696 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665710 5859 cpu_manager.go:414] "containerMap: removing container" podUID="945da409-b927-4d49-94cf-83a4655f5b44" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665715 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945da409-b927-4d49-94cf-83a4655f5b44" containerName="maas-api" Nov 02 21:55:13.665725 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665724 5859 cpu_manager.go:414] "containerMap: removing container" podUID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" containerName="maas-api" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665729 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" containerName="maas-api" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665738 5859 cpu_manager.go:414] "containerMap: removing container" podUID="4c436fcd-a610-4835-b495-016edb69349c" containerName="payload-pre-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665743 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c436fcd-a610-4835-b495-016edb69349c" containerName="payload-pre-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665754 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ee42edad-5982-436e-a25f-edd440443331" containerName="payload-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665759 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee42edad-5982-436e-a25f-edd440443331" containerName="payload-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665764 5859 cpu_manager.go:414] "containerMap: removing container" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665768 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665779 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665784 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665793 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665797 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665810 5859 cpu_manager.go:414] "containerMap: removing container" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665815 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665903 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee42edad-5982-436e-a25f-edd440443331" containerName="payload-processing" Nov 02 21:55:13.665908 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665910 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665916 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf9ba23-e654-412e-ba41-e96a4e054da7" containerName="payload-pre-processing" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665926 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8687284-6702-4573-a8d6-c5b5464ab9f6" containerName="istio-proxy" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665933 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c57a72-83b4-4ac8-9ed7-17ff0eba540f" containerName="istio-proxy" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665942 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b03d709-7126-4207-85a4-92a861f77f3c" containerName="maas-api" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665961 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eef6c43-6199-4e5c-9f26-5ccac44e59d5" containerName="maas-api" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665967 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="945da409-b927-4d49-94cf-83a4655f5b44" containerName="maas-api" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665974 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c4e8e2-94ad-49eb-b9ef-5821cc905aa0" containerName="maas-api" Nov 02 21:55:13.666397 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.665980 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c436fcd-a610-4835-b495-016edb69349c" containerName="payload-pre-processing" Nov 02 21:55:13.671047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.671027 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.761432 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.761385 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 21:55:13.793187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793130 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793385 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793292 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793466 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793408 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793470 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtwj\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793506 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793567 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793634 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793590 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793640 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.793745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.793664 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.804785 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.804697 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-58ca7f-openshift-default-dockercfg-89m6x\"" Nov 02 21:55:13.894548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894515 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894548 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894569 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtwj\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894601 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894657 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894681 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894798 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.894876 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894828 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.895287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.894901 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.895287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.895027 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.896287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.895722 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.896287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.895846 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.896287 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.896235 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.896856 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.896833 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.897239 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.897211 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.898417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.898391 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.905171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.905146 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.952205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.952146 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.952205 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.952169 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtwj\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\") pod \"e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:13.985297 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:13.985271 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:14.411527 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:14.411493 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 21:55:14.412495 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:55:14.412467 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5bf6ea_f433_4ed6_97f7_9dcfe45d1476.slice/crio-fcbc420af5860b3ea44e8780d736ed2e494fc65950404e0281a419b045f2de4e WatchSource:0}: Error finding container fcbc420af5860b3ea44e8780d736ed2e494fc65950404e0281a419b045f2de4e: Status 404 returned error can't find the container with id fcbc420af5860b3ea44e8780d736ed2e494fc65950404e0281a419b045f2de4e Nov 02 21:55:14.415711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:14.415680 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:14.415812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:14.415745 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:14.415812 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:14.415795 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:15.094219 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:15.094176 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerStarted","Data":"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6"} Nov 02 21:55:15.094654 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:15.094254 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerStarted","Data":"fcbc420af5860b3ea44e8780d736ed2e494fc65950404e0281a419b045f2de4e"} Nov 02 21:55:15.214628 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:15.214573 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podStartSLOduration=2.214560506 podStartE2EDuration="2.214560506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:15.210939825 +0000 UTC m=+2754.759189624" watchObservedRunningTime="2026-11-02 21:55:15.214560506 +0000 UTC m=+2754.762810288" Nov 02 21:55:15.986480 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:15.986439 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:15.991624 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:15.991598 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:16.102223 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:16.102194 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:16.103112 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:16.103093 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:55:25.946042 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:25.946005 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:55:29.773801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.773761 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 21:55:29.779040 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.779020 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.793506 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.793479 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc578\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.793648 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.793514 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.793734 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.793707 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.793790 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.793757 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.793920 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.793885 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.794082 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.794062 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.794169 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.794130 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.794232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.794163 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.794232 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.794218 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.832711 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.832681 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-06e461-openshift-default-dockercfg-xf4c7\"" Nov 02 21:55:29.886209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.886176 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 21:55:29.895831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.895797 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.895831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.895841 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.895886 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.895918 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc578\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.895943 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896089 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896126 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896119 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896193 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896401 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896342 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896677 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.896939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.896919 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.897117 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.897099 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.921047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.920984 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.921047 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.920998 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.921256 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.921186 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.921418 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.921398 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.953852 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.953814 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:29.954058 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:29.954035 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc578\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\") pod \"e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:30.096299 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.096272 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:30.525561 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.525530 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 21:55:30.527118 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:55:30.527092 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884afae4_99a7_4b0a_8fcf_bde5f08004d1.slice/crio-4f195a0054c590e97b751ede86b98511ee4be38939a737d0418f1c4ae211924b WatchSource:0}: Error finding container 4f195a0054c590e97b751ede86b98511ee4be38939a737d0418f1c4ae211924b: Status 404 returned error can't find the container with id 4f195a0054c590e97b751ede86b98511ee4be38939a737d0418f1c4ae211924b Nov 02 21:55:30.529724 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.529692 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:30.529801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.529758 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:30.529801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.529785 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:30.722894 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.722856 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 21:55:30.727141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.727116 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.802025 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.781662 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4ca09f-openshift-default-dockercfg-7mmpk\"" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817281 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817410 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817439 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817505 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817547 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817608 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hcv\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817638 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817687 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.817777 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.829971 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.827327 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 21:55:30.918780 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.918749 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.918988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.918820 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hcv\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.918988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.918856 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.918988 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.918904 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919021 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919166 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919086 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919187 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919212 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919274 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919218 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919417 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919316 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919521 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919488 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919591 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919666 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919637 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.919793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.919777 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.921449 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.921429 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.921673 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.921651 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.978145 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.978116 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hcv\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:30.978304 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:30.978122 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\") pod \"e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:31.044984 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.044922 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:31.208662 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.208625 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerStarted","Data":"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5"} Nov 02 21:55:31.208855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.208677 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerStarted","Data":"4f195a0054c590e97b751ede86b98511ee4be38939a737d0418f1c4ae211924b"} Nov 02 21:55:31.326722 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.326679 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podStartSLOduration=2.32666665 podStartE2EDuration="2.32666665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:31.32487356 +0000 UTC m=+2770.873123365" watchObservedRunningTime="2026-11-02 21:55:31.32666665 +0000 UTC m=+2770.874916453" Nov 02 21:55:31.449441 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.444335 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 21:55:31.467775 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:55:31.467744 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616867c4_1f91_4c94_a67a_61a52d9ee822.slice/crio-f4d19c4546b413fca5ead0e083260d6b2543e96448d4572cb384337aee4fa3d8 WatchSource:0}: Error finding container f4d19c4546b413fca5ead0e083260d6b2543e96448d4572cb384337aee4fa3d8: Status 404 returned error can't find the container with id f4d19c4546b413fca5ead0e083260d6b2543e96448d4572cb384337aee4fa3d8 Nov 02 21:55:31.470270 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.470237 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:31.470382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.470315 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:31.470382 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:31.470364 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:55:32.096924 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.096885 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:32.102592 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.102568 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:32.217408 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.217375 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerStarted","Data":"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659"} Nov 02 21:55:32.217559 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.217429 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerStarted","Data":"f4d19c4546b413fca5ead0e083260d6b2543e96448d4572cb384337aee4fa3d8"} Nov 02 21:55:32.217745 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.217722 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:32.218360 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.218341 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:55:32.453272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:32.453171 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podStartSLOduration=2.453158011 podStartE2EDuration="2.453158011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:32.451146236 +0000 UTC m=+2771.999396047" watchObservedRunningTime="2026-11-02 21:55:32.453158011 +0000 UTC m=+2772.001407791" Nov 02 21:55:33.046019 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:33.045992 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:33.051293 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:33.051267 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:33.223824 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:33.223787 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:33.224443 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:33.224421 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:55:51.861939 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:51.861903 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 21:55:51.866036 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:51.866019 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:51.920658 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:51.920626 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 21:55:51.966803 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:51.966777 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a1d751-serving-cert\"" Nov 02 21:55:52.009029 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.008995 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.009209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.009058 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.009209 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.009142 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9l6\" (UniqueName: \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.009410 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.009389 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.119984 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.120128 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.120182 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.120210 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9l6\" (UniqueName: \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:52.120713 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-serving-cert: secret "maas-api-e2e-shared-b-w3-a1d751-serving-cert" not found Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:52.120775 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls podName:d2779c71-cef4-4baa-9f78-a96efd64bdc4 nodeName:}" failed. No retries permitted until 2026-11-02 21:55:52.620755592 +0000 UTC m=+2792.169005367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls") pod "maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" (UID: "d2779c71-cef4-4baa-9f78-a96efd64bdc4") : secret "maas-api-e2e-shared-b-w3-a1d751-serving-cert" not found Nov 02 21:55:52.123057 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.121056 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.124392 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.124365 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.178027 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.177995 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9l6\" (UniqueName: \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.627979 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.627915 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.630354 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.630331 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:52.780933 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:52.780896 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:53.159712 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:53.159681 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 21:55:53.160638 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:55:53.160611 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2779c71_cef4_4baa_9f78_a96efd64bdc4.slice/crio-eb93f4c0a676684ab84b53fd73ad47193e72a6b1fd6cb7b8ffea4c2b2dfe9bab WatchSource:0}: Error finding container eb93f4c0a676684ab84b53fd73ad47193e72a6b1fd6cb7b8ffea4c2b2dfe9bab: Status 404 returned error can't find the container with id eb93f4c0a676684ab84b53fd73ad47193e72a6b1fd6cb7b8ffea4c2b2dfe9bab Nov 02 21:55:53.359425 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:53.359390 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" event={"ID":"d2779c71-cef4-4baa-9f78-a96efd64bdc4","Type":"ContainerStarted","Data":"eb93f4c0a676684ab84b53fd73ad47193e72a6b1fd6cb7b8ffea4c2b2dfe9bab"} Nov 02 21:55:54.369350 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:54.369300 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" event={"ID":"d2779c71-cef4-4baa-9f78-a96efd64bdc4","Type":"ContainerStarted","Data":"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384"} Nov 02 21:55:54.369805 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:54.369497 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:55:54.486850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:55:54.486787 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" podStartSLOduration=2.593835176 podStartE2EDuration="3.486772955s" totalImagesPullingTime="892.937779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:55:54.483699068 +0000 UTC m=+2794.031948867" watchObservedRunningTime="2026-11-02 21:55:54.486772955 +0000 UTC m=+2794.035022735" Nov 02 21:55:55.945920 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:55:55.945883 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:56:00.381121 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:00.381093 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 21:56:07.094980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.094896 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 21:56:07.104484 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.104467 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.153880 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.153848 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 21:56:07.201055 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.201034 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-0f0874-serving-cert\"" Nov 02 21:56:07.201426 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.201405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.201489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.201436 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.201489 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.201458 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kddk\" (UniqueName: \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.201631 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.201617 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.302767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.302736 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.302892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.302860 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.302892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.302881 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.303023 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.302901 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kddk\" (UniqueName: \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.303089 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:07.303031 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-serving-cert: secret "maas-api-e2e-shared-b-w0-0f0874-serving-cert" not found Nov 02 21:56:07.303145 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:07.303102 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls podName:ff48d4fe-28a8-48d7-9290-18bdd2483ea9 nodeName:}" failed. No retries permitted until 2026-11-02 21:56:07.803085307 +0000 UTC m=+2807.351335079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls") pod "maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" (UID: "ff48d4fe-28a8-48d7-9290-18bdd2483ea9") : secret "maas-api-e2e-shared-b-w0-0f0874-serving-cert" not found Nov 02 21:56:07.303545 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.303522 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.305102 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.305074 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.360347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.360296 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kddk\" (UniqueName: \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.810094 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.810008 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:07.812388 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:07.812360 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:08.017858 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:08.017832 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:08.390087 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:08.390055 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 21:56:08.390444 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:56:08.390229 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff48d4fe_28a8_48d7_9290_18bdd2483ea9.slice/crio-8b610f4b20fd717bd28ae374f0ba4b4a77674e4d2cf4fbf15b1c0d0c6331dac0 WatchSource:0}: Error finding container 8b610f4b20fd717bd28ae374f0ba4b4a77674e4d2cf4fbf15b1c0d0c6331dac0: Status 404 returned error can't find the container with id 8b610f4b20fd717bd28ae374f0ba4b4a77674e4d2cf4fbf15b1c0d0c6331dac0 Nov 02 21:56:08.474935 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:08.474912 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" event={"ID":"ff48d4fe-28a8-48d7-9290-18bdd2483ea9","Type":"ContainerStarted","Data":"8b610f4b20fd717bd28ae374f0ba4b4a77674e4d2cf4fbf15b1c0d0c6331dac0"} Nov 02 21:56:10.491186 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:10.491150 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" event={"ID":"ff48d4fe-28a8-48d7-9290-18bdd2483ea9","Type":"ContainerStarted","Data":"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680"} Nov 02 21:56:10.491614 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:10.491283 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:10.608285 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:10.608228 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" podStartSLOduration=2.5646467790000003 podStartE2EDuration="3.608214407s" totalImagesPullingTime="1.043567628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:10.606158051 +0000 UTC m=+2810.154407833" watchObservedRunningTime="2026-11-02 21:56:10.608214407 +0000 UTC m=+2810.156464188" Nov 02 21:56:16.503850 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:16.503819 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 21:56:25.945011 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:25.944976 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:56:37.583638 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.583607 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 21:56:37.588752 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.588736 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.642043 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.642017 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 21:56:37.669065 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.669036 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7p5\" (UniqueName: \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.669187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.669105 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.669187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.669133 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.669187 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.669151 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.689415 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.689394 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-06e461-serving-cert\"" Nov 02 21:56:37.770093 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.770068 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7p5\" (UniqueName: \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.770231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.770133 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.770231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.770165 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.770231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.770194 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.770378 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:37.770275 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-serving-cert: secret "maas-api-e2e-shared-a-w2-06e461-serving-cert" not found Nov 02 21:56:37.770378 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:37.770329 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls podName:72bc61a2-4e9d-4b20-959e-23dd4367d740 nodeName:}" failed. No retries permitted until 2026-11-02 21:56:38.270315776 +0000 UTC m=+2837.818565536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls") pod "maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" (UID: "72bc61a2-4e9d-4b20-959e-23dd4367d740") : secret "maas-api-e2e-shared-a-w2-06e461-serving-cert" not found Nov 02 21:56:37.770925 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.770904 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.772636 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.772614 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:37.828751 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:37.828726 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7p5\" (UniqueName: \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:38.276065 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:38.276029 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:38.278338 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:38.278313 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:38.501809 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:38.501780 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:38.875576 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:38.875548 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 21:56:38.876239 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:56:38.876210 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bc61a2_4e9d_4b20_959e_23dd4367d740.slice/crio-08a845c65227249911a75327e2f1b6664e5d953cde123094bb700353e7289afb WatchSource:0}: Error finding container 08a845c65227249911a75327e2f1b6664e5d953cde123094bb700353e7289afb: Status 404 returned error can't find the container with id 08a845c65227249911a75327e2f1b6664e5d953cde123094bb700353e7289afb Nov 02 21:56:39.666839 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:39.666801 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" event={"ID":"72bc61a2-4e9d-4b20-959e-23dd4367d740","Type":"ContainerStarted","Data":"08a845c65227249911a75327e2f1b6664e5d953cde123094bb700353e7289afb"} Nov 02 21:56:40.677152 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:40.677116 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" event={"ID":"72bc61a2-4e9d-4b20-959e-23dd4367d740","Type":"ContainerStarted","Data":"26fe81364d28e3d0545da9f0d5df0bff56d7a1673dcab32b77e855c836a6bacc"} Nov 02 21:56:40.677539 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:40.677202 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:40.793483 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:40.793436 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" podStartSLOduration=2.882827345 podStartE2EDuration="3.79342324s" totalImagesPullingTime="910.595895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:40.791421042 +0000 UTC m=+2840.339670823" watchObservedRunningTime="2026-11-02 21:56:40.79342324 +0000 UTC m=+2840.341673020" Nov 02 21:56:46.689142 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:56:46.689113 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 21:56:55.944784 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:56:55.944743 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:57:11.889932 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.889904 5859 generic.go:356] "Generic (PLEG): container finished" podID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerID="c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5" exitCode=0 Nov 02 21:57:11.890363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.889978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerDied","Data":"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5"} Nov 02 21:57:11.890363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.890311 5859 scope.go:122] "RemoveContainer" containerID="c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5" Nov 02 21:57:11.891736 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.891712 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:11.891796 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.891774 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:11.891831 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:11.891808 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:12.903519 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:12.903479 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerStarted","Data":"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb"} Nov 02 21:57:13.096892 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:13.096858 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:57:13.098177 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:13.098153 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:13.098303 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:13.098198 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:14.097149 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:14.097104 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:14.097655 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:14.097174 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:15.097231 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:15.097198 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:15.097598 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:15.097249 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:16.096649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:16.096614 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:16.096814 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:16.096674 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:17.096929 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:17.096894 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:17.097340 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:17.096977 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:17.226068 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:17.226031 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:57:18.096727 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:18.096692 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 02 21:57:18.096900 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:18.096746 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 02 21:57:19.101188 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:19.101159 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:57:19.102272 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:19.102249 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 21:57:25.943818 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:57:25.943784 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:57:44.178072 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.178008 5859 generic.go:356] "Generic (PLEG): container finished" podID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerID="058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6" exitCode=0 Nov 02 21:57:44.178453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.178082 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerDied","Data":"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6"} Nov 02 21:57:44.178453 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.178416 5859 scope.go:122] "RemoveContainer" containerID="058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6" Nov 02 21:57:44.179916 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.179886 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:44.180044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.179966 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:44.180044 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:44.180005 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:57:45.185961 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:45.185915 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerStarted","Data":"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf"} Nov 02 21:57:45.986192 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:45.986161 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:57:45.987198 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:45.987171 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 21:57:45.987327 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:45.987220 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 21:57:46.102766 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:46.102728 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:57:46.986260 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:46.986223 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 21:57:46.986677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:46.986308 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 21:57:47.986085 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:47.986052 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 21:57:47.986259 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:47.986100 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 21:57:48.985755 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:48.985731 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 21:57:48.986157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:48.985782 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 21:57:49.986180 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:49.986146 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 21:57:49.986588 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:49.986202 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 21:57:50.989967 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:50.989920 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:57:50.991168 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:57:50.991151 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 21:57:55.948801 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:57:55.948735 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:58:10.401589 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.401528 5859 generic.go:356] "Generic (PLEG): container finished" podID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerID="b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659" exitCode=0 Nov 02 21:58:10.402038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.401596 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerDied","Data":"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659"} Nov 02 21:58:10.402038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.402006 5859 scope.go:122] "RemoveContainer" containerID="b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659" Nov 02 21:58:10.403430 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.403399 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:10.403513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.403470 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:10.403513 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:10.403502 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:58:11.436261 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:11.436225 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerStarted","Data":"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc"} Nov 02 21:58:12.046105 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:12.046071 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:58:12.046296 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:12.046262 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:58:12.046345 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:12.046320 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 21:58:13.045525 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:13.045491 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:58:13.046049 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:13.045557 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 21:58:14.045765 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:14.045731 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:58:14.046182 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:14.045781 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 21:58:15.045515 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:15.045479 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:58:15.045754 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:15.045543 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 21:58:16.045946 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:16.045916 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 21:58:16.046337 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:16.045986 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 21:58:17.049995 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:17.049966 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:58:17.050381 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:17.050204 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:58:17.050801 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:17.050785 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 21:58:25.943541 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:25.943509 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:58:49.047462 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.047423 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:58:49.048056 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.047609 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" containerID="cri-o://922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a" gracePeriod=30 Nov 02 21:58:49.097899 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.097868 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:58:49.098113 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.098089 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" containerID="cri-o://0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d" gracePeriod=30 Nov 02 21:58:49.100841 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.100816 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:58:49.101038 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.101015 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" containerID="cri-o://28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1" gracePeriod=30 Nov 02 21:58:49.162100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.162064 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:58:49.162399 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.162377 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:58:49.162802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.162748 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" containerID="cri-o://9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7" gracePeriod=30 Nov 02 21:58:49.163235 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:49.163207 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" containerID="cri-o://f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb" gracePeriod=30 Nov 02 21:58:54.387992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.387944 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:58:54.433579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.433397 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:58:54.433579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.433456 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:58:54.481862 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.481835 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:58:54.494992 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.488085 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:58:54.534322 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534296 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534340 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534464 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534450 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534511 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534579 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534553 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534571 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.534679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534591 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534625 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534679 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534659 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534698 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\") pod \"de4eb182-c5e9-400e-8581-5204f8b48ed2\" (UID: \"de4eb182-c5e9-400e-8581-5204f8b48ed2\") " Nov 02 21:58:54.534926 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.534891 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535162 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535525 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535546 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535563 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4eb182-c5e9-400e-8581-5204f8b48ed2-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535753 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.535851 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.535685 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.538171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.538068 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "kube-api-access-f8sjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.538171 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.538131 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.538478 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.538450 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.538793 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.538766 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token" pod "de4eb182-c5e9-400e-8581-5204f8b48ed2" (UID: "de4eb182-c5e9-400e-8581-5204f8b48ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.545034 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.544942 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:58:54.549891 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.549867 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:58:54.636294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636214 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.636294 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636261 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.636552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636316 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.636552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636350 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.636552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636407 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.636552 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636454 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636552 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636614 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636657 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636690 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636729 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636778 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636804 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636835 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636862 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636888 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636924 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.636988 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.637015 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637023 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637064 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637094 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637120 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637159 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637177 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637200 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\") pod \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\" (UID: \"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637239 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637270 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637297 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637306 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637418 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637490 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637541 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637578 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637609 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637641 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.637689 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637668 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\") pod \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\" (UID: \"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c\") " Nov 02 21:58:54.638626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637720 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\") pod \"0c5bace8-1b13-476c-8056-ca50b3150a4b\" (UID: \"0c5bace8-1b13-476c-8056-ca50b3150a4b\") " Nov 02 21:58:54.638626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.637754 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\") pod \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\" (UID: \"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f\") " Nov 02 21:58:54.638626 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638486 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.638767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638743 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638767 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638769 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sjd\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-kube-api-access-f8sjd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638785 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638800 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638814 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638827 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638842 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638856 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.638986 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.638872 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4eb182-c5e9-400e-8581-5204f8b48ed2-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.639387 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.639356 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.639446 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.639407 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.639872 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.639848 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.640254 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.640213 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.640904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.640312 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.640904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.640518 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.640904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.640575 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.640904 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.640796 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.641556 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.641511 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.641556 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.641549 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.641795 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.641876 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642196 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642404 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "kube-api-access-v4gjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642595 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642656 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "kube-api-access-qstdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642875 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.642916 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.643069 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.643163 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.643133 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.644317 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.643245 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.644864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.644756 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.644864 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.644836 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.645158 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.645132 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.647593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.647500 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.647593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.647511 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.647593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.647543 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447" pod "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" (UID: "b477b6e7-5c7e-4e33-ac12-0b02bc5de97f"). InnerVolumeSpecName "kube-api-access-ll447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.647593 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.647572 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.647857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.647838 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.648597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.648425 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.648597 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.648544 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token" pod "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" (UID: "d3b8fafd-3e6b-4a59-9836-a022f9f1be8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.649141 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.649118 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo" pod "0c5bace8-1b13-476c-8056-ca50b3150a4b" (UID: "0c5bace8-1b13-476c-8056-ca50b3150a4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.650258 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.650234 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh" pod "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" (UID: "0c3edf2e-1ee7-48dc-90d9-c8895bc0212c"). InnerVolumeSpecName "kube-api-access-w5zwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.740328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740299 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740328 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740326 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740338 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740353 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740367 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740378 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5bace8-1b13-476c-8056-ca50b3150a4b-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740392 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740404 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740416 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740428 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gjm\" (UniqueName: \"kubernetes.io/projected/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-kube-api-access-v4gjm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740442 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740454 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740465 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740477 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740489 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740502 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740517 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qstdw\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-kube-api-access-qstdw\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740532 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zwh\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-kube-api-access-w5zwh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740546 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.740555 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740558 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740572 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740585 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll447\" (UniqueName: \"kubernetes.io/projected/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-kube-api-access-ll447\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740599 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740613 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740625 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740639 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740654 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740666 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740679 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5bace8-1b13-476c-8056-ca50b3150a4b-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740693 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740706 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740720 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.741262 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.740733 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.747639 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.747616 5859 generic.go:356] "Generic (PLEG): container finished" podID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerID="28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1" exitCode=0 Nov 02 21:58:54.747737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.747665 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" Nov 02 21:58:54.747737 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.747702 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" event={"ID":"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c","Type":"ContainerDied","Data":"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1"} Nov 02 21:58:54.747857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.747765 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" event={"ID":"0c3edf2e-1ee7-48dc-90d9-c8895bc0212c","Type":"ContainerDied","Data":"cbc4771a040287934bf2bda4868c588b957293b5bd619530a65f26ccfb50e200"} Nov 02 21:58:54.747857 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.747797 5859 scope.go:122] "RemoveContainer" containerID="28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1" Nov 02 21:58:54.749116 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.749095 5859 generic.go:356] "Generic (PLEG): container finished" podID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerID="922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a" exitCode=0 Nov 02 21:58:54.749206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.749142 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" Nov 02 21:58:54.749206 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.749180 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" event={"ID":"de4eb182-c5e9-400e-8581-5204f8b48ed2","Type":"ContainerDied","Data":"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a"} Nov 02 21:58:54.749319 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.749238 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx" event={"ID":"de4eb182-c5e9-400e-8581-5204f8b48ed2","Type":"ContainerDied","Data":"5f75265d921bfd3ab8cb5cd92957cbe7d676cac39d38a9fe673e40ed017bf1c1"} Nov 02 21:58:54.750698 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.750680 5859 generic.go:356] "Generic (PLEG): container finished" podID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerID="9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7" exitCode=0 Nov 02 21:58:54.751225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.750731 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" Nov 02 21:58:54.751225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.750759 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" event={"ID":"0c5bace8-1b13-476c-8056-ca50b3150a4b","Type":"ContainerDied","Data":"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7"} Nov 02 21:58:54.751225 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.750823 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2" event={"ID":"0c5bace8-1b13-476c-8056-ca50b3150a4b","Type":"ContainerDied","Data":"7f39cec1fc41f44f34a22a8ab81b8f65a9a61b8ee28b1e9155235b43ce31112a"} Nov 02 21:58:54.752501 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.752480 5859 generic.go:356] "Generic (PLEG): container finished" podID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerID="0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d" exitCode=0 Nov 02 21:58:54.752594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.752529 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" Nov 02 21:58:54.752594 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.752567 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" event={"ID":"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b","Type":"ContainerDied","Data":"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d"} Nov 02 21:58:54.752708 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.752610 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg" event={"ID":"d3b8fafd-3e6b-4a59-9836-a022f9f1be8b","Type":"ContainerDied","Data":"0bfe855fda80f55138de4247a4571b44afbad7a4b82b6bc00cc572135eb5193b"} Nov 02 21:58:54.754213 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.754189 5859 generic.go:356] "Generic (PLEG): container finished" podID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerID="f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb" exitCode=0 Nov 02 21:58:54.754320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.754268 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" Nov 02 21:58:54.754320 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.754276 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" event={"ID":"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f","Type":"ContainerDied","Data":"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb"} Nov 02 21:58:54.754421 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.754329 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv" event={"ID":"b477b6e7-5c7e-4e33-ac12-0b02bc5de97f","Type":"ContainerDied","Data":"0e00ef50fc9d8a897de4ba0532e46173ab936b3d6a8059f6980962f382bb049d"} Nov 02 21:58:54.761265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.759830 5859 scope.go:122] "RemoveContainer" containerID="28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1" Nov 02 21:58:54.761265 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:54.760181 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1\": container with ID starting with 28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1 not found: ID does not exist" containerID="28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1" Nov 02 21:58:54.761265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.760211 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1"} err="failed to get container status \"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1\": rpc error: code = NotFound desc = could not find container \"28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1\": container with ID starting with 28fc1e41be54cacb864dabf0700dfaaa81b6b6ffa40743504ebaaaa8dfd606d1 not found: ID does not exist" Nov 02 21:58:54.761265 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.760236 5859 scope.go:122] "RemoveContainer" containerID="922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a" Nov 02 21:58:54.772343 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.772328 5859 scope.go:122] "RemoveContainer" containerID="922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a" Nov 02 21:58:54.772583 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:54.772564 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a\": container with ID starting with 922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a not found: ID does not exist" containerID="922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a" Nov 02 21:58:54.772649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.772592 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a"} err="failed to get container status \"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a\": rpc error: code = NotFound desc = could not find container \"922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a\": container with ID starting with 922d5ea90cb41beb6c0151007be72023a1da5b88ef2e3c5823c5ceebf7a9684a not found: ID does not exist" Nov 02 21:58:54.772649 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.772614 5859 scope.go:122] "RemoveContainer" containerID="9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7" Nov 02 21:58:54.791276 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.791251 5859 scope.go:122] "RemoveContainer" containerID="9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7" Nov 02 21:58:54.791731 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:54.791694 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7\": container with ID starting with 9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7 not found: ID does not exist" containerID="9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7" Nov 02 21:58:54.791848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.791740 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7"} err="failed to get container status \"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7\": rpc error: code = NotFound desc = could not find container \"9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7\": container with ID starting with 9528bef1e1ca5b14551c147a26c1a12711ccf1140be3115fe081666f9f06b7b7 not found: ID does not exist" Nov 02 21:58:54.791848 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.791762 5859 scope.go:122] "RemoveContainer" containerID="0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d" Nov 02 21:58:54.802855 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.802839 5859 scope.go:122] "RemoveContainer" containerID="0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d" Nov 02 21:58:54.803148 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:54.803118 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d\": container with ID starting with 0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d not found: ID does not exist" containerID="0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d" Nov 02 21:58:54.803261 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.803156 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d"} err="failed to get container status \"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d\": rpc error: code = NotFound desc = could not find container \"0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d\": container with ID starting with 0c0c8204be2df403a703dc316e440bc84dd40eda7b23f7f20b078a2eac17c27d not found: ID does not exist" Nov 02 21:58:54.803261 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.803178 5859 scope.go:122] "RemoveContainer" containerID="f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb" Nov 02 21:58:54.812398 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.812382 5859 scope.go:122] "RemoveContainer" containerID="f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb" Nov 02 21:58:54.812637 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:54.812620 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb\": container with ID starting with f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb not found: ID does not exist" containerID="f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb" Nov 02 21:58:54.812700 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:54.812643 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb"} err="failed to get container status \"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb\": rpc error: code = NotFound desc = could not find container \"f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb\": container with ID starting with f0fac1ceb55f6e00619fec958b8e0b19211d24d6aca44aef0bdf89613c2fe9fb not found: ID does not exist" Nov 02 21:58:55.138348 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.138314 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:58:55.141869 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.141846 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-bd2fg"] Nov 02 21:58:55.251005 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.250974 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:58:55.254442 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.254418 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj"] Nov 02 21:58:55.363321 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.363261 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:58:55.366677 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.366656 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jzqq2"] Nov 02 21:58:55.476735 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.476700 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:58:55.476735 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.476725 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:58:55.477157 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.476748 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-z5hwj" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:58:55.482935 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.482908 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-7w5sx"] Nov 02 21:58:55.592846 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.592823 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:58:55.596313 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:55.596293 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-ftblv"] Nov 02 21:58:55.947985 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:58:55.946821 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:58:56.570509 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:56.570466 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" path="/var/lib/kubelet/pods/0c3edf2e-1ee7-48dc-90d9-c8895bc0212c/volumes" Nov 02 21:58:56.571114 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:56.571091 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" path="/var/lib/kubelet/pods/0c5bace8-1b13-476c-8056-ca50b3150a4b/volumes" Nov 02 21:58:56.571585 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:56.571564 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" path="/var/lib/kubelet/pods/b477b6e7-5c7e-4e33-ac12-0b02bc5de97f/volumes" Nov 02 21:58:56.572363 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:56.572343 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" path="/var/lib/kubelet/pods/d3b8fafd-3e6b-4a59-9836-a022f9f1be8b/volumes" Nov 02 21:58:56.572802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:58:56.572783 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" path="/var/lib/kubelet/pods/de4eb182-c5e9-400e-8581-5204f8b48ed2/volumes" Nov 02 21:59:17.633283 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.633200 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 21:59:17.634014 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.633996 5859 cpu_manager.go:414] "containerMap: removing container" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" Nov 02 21:59:17.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634017 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" Nov 02 21:59:17.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634037 5859 cpu_manager.go:414] "containerMap: removing container" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" Nov 02 21:59:17.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634044 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" Nov 02 21:59:17.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634055 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" Nov 02 21:59:17.634070 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634062 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" Nov 02 21:59:17.634221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634077 5859 cpu_manager.go:414] "containerMap: removing container" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" Nov 02 21:59:17.634221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634084 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" Nov 02 21:59:17.634221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634111 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" Nov 02 21:59:17.634221 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634118 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" Nov 02 21:59:17.634341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634242 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b8fafd-3e6b-4a59-9836-a022f9f1be8b" containerName="istio-proxy" Nov 02 21:59:17.634341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634262 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5bace8-1b13-476c-8056-ca50b3150a4b" containerName="istio-proxy" Nov 02 21:59:17.634341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634271 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3edf2e-1ee7-48dc-90d9-c8895bc0212c" containerName="istio-proxy" Nov 02 21:59:17.634341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634277 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b477b6e7-5c7e-4e33-ac12-0b02bc5de97f" containerName="istio-proxy" Nov 02 21:59:17.634341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.634287 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4eb182-c5e9-400e-8581-5204f8b48ed2" containerName="istio-proxy" Nov 02 21:59:17.640139 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.640118 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.691447 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.691419 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 21:59:17.737430 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.737408 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c7e71f-serving-cert\"" Nov 02 21:59:17.764542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.764523 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.764648 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.764560 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.764697 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.764685 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.764802 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.764778 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkf5\" (UniqueName: \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.865368 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.865343 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.865549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.865385 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.865549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.865434 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.865549 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.865512 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkf5\" (UniqueName: \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.865716 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:59:17.865601 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-serving-cert: secret "maas-api-e2e-shared-b-w2-c7e71f-serving-cert" not found Nov 02 21:59:17.865716 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:59:17.865673 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls podName:ac857156-a817-4303-a981-5c741136a896 nodeName:}" failed. No retries permitted until 2026-11-02 21:59:18.3656584 +0000 UTC m=+2997.913908159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" (UID: "ac857156-a817-4303-a981-5c741136a896") : secret "maas-api-e2e-shared-b-w2-c7e71f-serving-cert" not found Nov 02 21:59:17.866100 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.866077 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.867672 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.867651 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:17.923406 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:17.923334 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkf5\" (UniqueName: \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:18.370660 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:18.370622 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:18.372980 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:18.372942 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:18.553887 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:18.553858 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:18.951341 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:18.951310 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 21:59:18.951991 ip-10-0-137-1 kubenswrapper[5859]: W1102 21:59:18.951945 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac857156_a817_4303_a981_5c741136a896.slice/crio-dcc015a21ce4ccaaa3fd62b290d4ec11831c68457bf02e91f7afe2eb851455f9 WatchSource:0}: Error finding container dcc015a21ce4ccaaa3fd62b290d4ec11831c68457bf02e91f7afe2eb851455f9: Status 404 returned error can't find the container with id dcc015a21ce4ccaaa3fd62b290d4ec11831c68457bf02e91f7afe2eb851455f9 Nov 02 21:59:18.953347 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:18.953328 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:59:19.914542 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:19.914509 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" event={"ID":"ac857156-a817-4303-a981-5c741136a896","Type":"ContainerStarted","Data":"dcc015a21ce4ccaaa3fd62b290d4ec11831c68457bf02e91f7afe2eb851455f9"} Nov 02 21:59:20.922288 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:20.922252 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" event={"ID":"ac857156-a817-4303-a981-5c741136a896","Type":"ContainerStarted","Data":"9fe16f2e1e71d987ab596e9eff818a5e42f835fa361f6056772bfaf507c43b51"} Nov 02 21:59:20.922758 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:20.922313 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:21.038938 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:21.038885 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" podStartSLOduration=3.067195106 podStartE2EDuration="4.038872046s" totalImagesPullingTime="971.67694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:21.0371506 +0000 UTC m=+3000.585400380" watchObservedRunningTime="2026-11-02 21:59:21.038872046 +0000 UTC m=+3000.587121826" Nov 02 21:59:25.944713 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:59:25.944680 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 21:59:26.934208 ip-10-0-137-1 kubenswrapper[5859]: I1102 21:59:26.934183 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 21:59:55.944977 ip-10-0-137-1 kubenswrapper[5859]: E1102 21:59:55.944929 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:00:00.204721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.204639 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:00.209781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.209747 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:00.209933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.209853 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.213989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.213971 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.223863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.223844 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:00.227776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.227759 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.290941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.290908 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309636 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309697 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmk4v\" (UniqueName: \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309746 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309774 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309830 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqm6s\" (UniqueName: \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.311716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.309943 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hwk\" (UniqueName: \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.322153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.320888 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:00.324034 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.324005 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:00.366431 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.366412 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:00:00.411214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411183 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmk4v\" (UniqueName: \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.411338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411233 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.411338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411322 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.411454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411360 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqm6s\" (UniqueName: \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.411454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411436 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hwk\" (UniqueName: \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.411558 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411522 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.411947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411925 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.412051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.411991 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.412134 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.412115 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.468374 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.468324 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqm6s\" (UniqueName: \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\") pod \"maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.468461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.468389 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hwk\" (UniqueName: \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.468601 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.468580 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmk4v\" (UniqueName: \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.524764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.524741 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:00.542731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.542707 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:00.550865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.550841 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:00.904850 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.904826 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:00.906121 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:00:00.906103 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c6d2df_db0c_4006_b15e_31dda383a373.slice/crio-a96053af78daf48adb97352b2f302fdfa78c535b74056ece0dcef322e11a5e67 WatchSource:0}: Error finding container a96053af78daf48adb97352b2f302fdfa78c535b74056ece0dcef322e11a5e67: Status 404 returned error can't find the container with id a96053af78daf48adb97352b2f302fdfa78c535b74056ece0dcef322e11a5e67 Nov 02 22:00:00.952515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.952490 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:00.953390 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:00:00.953365 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65672911_c370_457c_b393_c4d341613524.slice/crio-607c388e6c125b37af995a8d4207f1a037ef25473cda93f247c14782f67645fe WatchSource:0}: Error finding container 607c388e6c125b37af995a8d4207f1a037ef25473cda93f247c14782f67645fe: Status 404 returned error can't find the container with id 607c388e6c125b37af995a8d4207f1a037ef25473cda93f247c14782f67645fe Nov 02 22:00:00.960277 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:00.956460 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:00.960277 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:00:00.960058 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5367fbb7_b30d_418b_922c_7012d0102dea.slice/crio-9f3e0c88cedb4cb1deb382d4657996841a3ac682f3d5836e10f1b86422807a35 WatchSource:0}: Error finding container 9f3e0c88cedb4cb1deb382d4657996841a3ac682f3d5836e10f1b86422807a35: Status 404 returned error can't find the container with id 9f3e0c88cedb4cb1deb382d4657996841a3ac682f3d5836e10f1b86422807a35 Nov 02 22:00:01.173271 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:01.173199 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerStarted","Data":"9f3e0c88cedb4cb1deb382d4657996841a3ac682f3d5836e10f1b86422807a35"} Nov 02 22:00:01.174483 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:01.174458 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerStarted","Data":"a96053af78daf48adb97352b2f302fdfa78c535b74056ece0dcef322e11a5e67"} Nov 02 22:00:01.175739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:01.175715 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerStarted","Data":"607c388e6c125b37af995a8d4207f1a037ef25473cda93f247c14782f67645fe"} Nov 02 22:00:02.183919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.183889 5859 generic.go:356] "Generic (PLEG): container finished" podID="5367fbb7-b30d-418b-922c-7012d0102dea" containerID="c708f37c1ca518d918a53ef8dd7feb13a077a553b3eaa1b21dc74f8465144400" exitCode=60 Nov 02 22:00:02.184379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.183978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerDied","Data":"c708f37c1ca518d918a53ef8dd7feb13a077a553b3eaa1b21dc74f8465144400"} Nov 02 22:00:02.184379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.184249 5859 scope.go:122] "RemoveContainer" containerID="c708f37c1ca518d918a53ef8dd7feb13a077a553b3eaa1b21dc74f8465144400" Nov 02 22:00:02.185833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.185708 5859 generic.go:356] "Generic (PLEG): container finished" podID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerID="7526481ddfe5db71e2361763a8c4e47e0fe2cf72ecb3552bf30cae7fa1eb2d92" exitCode=60 Nov 02 22:00:02.185908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.185834 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerDied","Data":"7526481ddfe5db71e2361763a8c4e47e0fe2cf72ecb3552bf30cae7fa1eb2d92"} Nov 02 22:00:02.186017 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.186001 5859 scope.go:122] "RemoveContainer" containerID="7526481ddfe5db71e2361763a8c4e47e0fe2cf72ecb3552bf30cae7fa1eb2d92" Nov 02 22:00:02.187532 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.187507 5859 generic.go:356] "Generic (PLEG): container finished" podID="65672911-c370-457c-b393-c4d341613524" containerID="7950497e72b09b6361c4774347f2d8d6934f320d9550326388359be9ebd0c78d" exitCode=60 Nov 02 22:00:02.187586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.187540 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerDied","Data":"7950497e72b09b6361c4774347f2d8d6934f320d9550326388359be9ebd0c78d"} Nov 02 22:00:02.187873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:02.187858 5859 scope.go:122] "RemoveContainer" containerID="7950497e72b09b6361c4774347f2d8d6934f320d9550326388359be9ebd0c78d" Nov 02 22:00:03.195839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.195807 5859 generic.go:356] "Generic (PLEG): container finished" podID="65672911-c370-457c-b393-c4d341613524" containerID="68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460" exitCode=60 Nov 02 22:00:03.196251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.195931 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerDied","Data":"68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460"} Nov 02 22:00:03.196251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.196013 5859 scope.go:122] "RemoveContainer" containerID="7950497e72b09b6361c4774347f2d8d6934f320d9550326388359be9ebd0c78d" Nov 02 22:00:03.196251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.196190 5859 scope.go:122] "RemoveContainer" containerID="68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460" Nov 02 22:00:03.196530 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:03.196500 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479_odh-ai-gateway-infra(65672911-c370-457c-b393-c4d341613524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" podUID="65672911-c370-457c-b393-c4d341613524" Nov 02 22:00:03.201728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.201558 5859 generic.go:356] "Generic (PLEG): container finished" podID="5367fbb7-b30d-418b-922c-7012d0102dea" containerID="961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0" exitCode=60 Nov 02 22:00:03.201848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.201749 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerDied","Data":"961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0"} Nov 02 22:00:03.202116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.202096 5859 scope.go:122] "RemoveContainer" containerID="961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0" Nov 02 22:00:03.202451 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:03.202413 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp_odh-ai-gateway-infra(5367fbb7-b30d-418b-922c-7012d0102dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" Nov 02 22:00:03.210039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.206997 5859 generic.go:356] "Generic (PLEG): container finished" podID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerID="1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862" exitCode=60 Nov 02 22:00:03.210039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.207104 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerDied","Data":"1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862"} Nov 02 22:00:03.210039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.207487 5859 scope.go:122] "RemoveContainer" containerID="1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862" Nov 02 22:00:03.210039 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:03.207787 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf_odh-ai-gateway-infra(f1c6d2df-db0c-4006-b15e-31dda383a373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" Nov 02 22:00:03.321896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.321878 5859 scope.go:122] "RemoveContainer" containerID="c708f37c1ca518d918a53ef8dd7feb13a077a553b3eaa1b21dc74f8465144400" Nov 02 22:00:03.334810 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:03.334788 5859 scope.go:122] "RemoveContainer" containerID="7526481ddfe5db71e2361763a8c4e47e0fe2cf72ecb3552bf30cae7fa1eb2d92" Nov 02 22:00:04.214941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:04.214913 5859 scope.go:122] "RemoveContainer" containerID="1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862" Nov 02 22:00:04.215453 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:04.215173 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf_odh-ai-gateway-infra(f1c6d2df-db0c-4006-b15e-31dda383a373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" Nov 02 22:00:04.216834 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:04.216811 5859 scope.go:122] "RemoveContainer" containerID="68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460" Nov 02 22:00:04.217079 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:04.217056 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479_odh-ai-gateway-infra(65672911-c370-457c-b393-c4d341613524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" podUID="65672911-c370-457c-b393-c4d341613524" Nov 02 22:00:04.218353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:04.218333 5859 scope.go:122] "RemoveContainer" containerID="961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0" Nov 02 22:00:04.218519 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:04.218502 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp_odh-ai-gateway-infra(5367fbb7-b30d-418b-922c-7012d0102dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" Nov 02 22:00:13.564033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:13.564004 5859 scope.go:122] "RemoveContainer" containerID="961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0" Nov 02 22:00:14.295199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:14.295109 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerStarted","Data":"89d8b717e6ad14da7689e813b87b8621a271052ede08039d6d5c3bb55310015d"} Nov 02 22:00:14.409539 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:14.409497 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" podStartSLOduration=14.409485188 podStartE2EDuration="14.409485188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:14.408133591 +0000 UTC m=+3053.956383390" watchObservedRunningTime="2026-11-02 22:00:14.409485188 +0000 UTC m=+3053.957734968" Nov 02 22:00:15.303036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.303005 5859 generic.go:356] "Generic (PLEG): container finished" podID="5367fbb7-b30d-418b-922c-7012d0102dea" containerID="89d8b717e6ad14da7689e813b87b8621a271052ede08039d6d5c3bb55310015d" exitCode=60 Nov 02 22:00:15.303508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.303077 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerDied","Data":"89d8b717e6ad14da7689e813b87b8621a271052ede08039d6d5c3bb55310015d"} Nov 02 22:00:15.303508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.303146 5859 scope.go:122] "RemoveContainer" containerID="961ebe0ee3b88958b72fdebd8b8fb8ca0f0f74e754b91bb47c43c81166a446d0" Nov 02 22:00:15.303508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.303395 5859 scope.go:122] "RemoveContainer" containerID="89d8b717e6ad14da7689e813b87b8621a271052ede08039d6d5c3bb55310015d" Nov 02 22:00:15.303665 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:15.303612 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp_odh-ai-gateway-infra(5367fbb7-b30d-418b-922c-7012d0102dea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" Nov 02 22:00:15.419896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.419854 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:15.563707 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:15.563645 5859 scope.go:122] "RemoveContainer" containerID="1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862" Nov 02 22:00:16.314859 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.314827 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerStarted","Data":"b0186b0a7df89fde4c0253d9fc2bdf6e5d5c35ca345a543b71a37ece575c9340"} Nov 02 22:00:16.481977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.481934 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:16.539375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.539337 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" podStartSLOduration=16.539324442 podStartE2EDuration="16.539324442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:16.538305064 +0000 UTC m=+3056.086554844" watchObservedRunningTime="2026-11-02 22:00:16.539324442 +0000 UTC m=+3056.087574238" Nov 02 22:00:16.563245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.563197 5859 scope.go:122] "RemoveContainer" containerID="68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460" Nov 02 22:00:16.591239 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.591208 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\") pod \"5367fbb7-b30d-418b-922c-7012d0102dea\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " Nov 02 22:00:16.591364 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.591273 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\" (UniqueName: \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\") pod \"5367fbb7-b30d-418b-922c-7012d0102dea\" (UID: \"5367fbb7-b30d-418b-922c-7012d0102dea\") " Nov 02 22:00:16.591578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.591552 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca" pod "5367fbb7-b30d-418b-922c-7012d0102dea" (UID: "5367fbb7-b30d-418b-922c-7012d0102dea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:16.591717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.591702 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5367fbb7-b30d-418b-922c-7012d0102dea-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:16.593398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.593378 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s" pod "5367fbb7-b30d-418b-922c-7012d0102dea" (UID: "5367fbb7-b30d-418b-922c-7012d0102dea"). InnerVolumeSpecName "kube-api-access-qqm6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:16.693019 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:16.692992 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqm6s\" (UniqueName: \"kubernetes.io/projected/5367fbb7-b30d-418b-922c-7012d0102dea-kube-api-access-qqm6s\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.322693 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.322666 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" Nov 02 22:00:17.323188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.322665 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp" event={"ID":"5367fbb7-b30d-418b-922c-7012d0102dea","Type":"ContainerDied","Data":"9f3e0c88cedb4cb1deb382d4657996841a3ac682f3d5836e10f1b86422807a35"} Nov 02 22:00:17.323188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.322814 5859 scope.go:122] "RemoveContainer" containerID="89d8b717e6ad14da7689e813b87b8621a271052ede08039d6d5c3bb55310015d" Nov 02 22:00:17.325824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.325801 5859 generic.go:356] "Generic (PLEG): container finished" podID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerID="b0186b0a7df89fde4c0253d9fc2bdf6e5d5c35ca345a543b71a37ece575c9340" exitCode=60 Nov 02 22:00:17.326004 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.325868 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerDied","Data":"b0186b0a7df89fde4c0253d9fc2bdf6e5d5c35ca345a543b71a37ece575c9340"} Nov 02 22:00:17.326333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.326312 5859 scope.go:122] "RemoveContainer" containerID="b0186b0a7df89fde4c0253d9fc2bdf6e5d5c35ca345a543b71a37ece575c9340" Nov 02 22:00:17.326609 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:17.326587 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf_odh-ai-gateway-infra(f1c6d2df-db0c-4006-b15e-31dda383a373)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" Nov 02 22:00:17.328312 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.328290 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerStarted","Data":"92c1bc9ffc2991c95d1760fa596c0b44448fe1d0b7cdba1dbdc277354d5cf185"} Nov 02 22:00:17.334451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.334435 5859 scope.go:122] "RemoveContainer" containerID="1177b5d1e5ca0f42bb97f3a4943b9e89fc54de6e23368c46d0eea2f842590862" Nov 02 22:00:17.548935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.548904 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:17.605101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.605003 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" podStartSLOduration=17.60499077 podStartE2EDuration="17.60499077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:17.602474017 +0000 UTC m=+3057.150723797" watchObservedRunningTime="2026-11-02 22:00:17.60499077 +0000 UTC m=+3057.153240550" Nov 02 22:00:17.714990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.714941 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:17.719680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:17.719653 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-4ca09f-29894280-wl9sp"] Nov 02 22:00:18.337745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.337717 5859 generic.go:356] "Generic (PLEG): container finished" podID="65672911-c370-457c-b393-c4d341613524" containerID="92c1bc9ffc2991c95d1760fa596c0b44448fe1d0b7cdba1dbdc277354d5cf185" exitCode=60 Nov 02 22:00:18.338185 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.337785 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerDied","Data":"92c1bc9ffc2991c95d1760fa596c0b44448fe1d0b7cdba1dbdc277354d5cf185"} Nov 02 22:00:18.338185 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.337844 5859 scope.go:122] "RemoveContainer" containerID="68d843484b79597c4469f48a90afe3e33381976aa400b5e0ed62bee3e1c49460" Nov 02 22:00:18.338283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.338243 5859 scope.go:122] "RemoveContainer" containerID="92c1bc9ffc2991c95d1760fa596c0b44448fe1d0b7cdba1dbdc277354d5cf185" Nov 02 22:00:18.338605 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:18.338570 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479_odh-ai-gateway-infra(65672911-c370-457c-b393-c4d341613524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" podUID="65672911-c370-457c-b393-c4d341613524" Nov 02 22:00:18.492911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.492878 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:18.570503 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.570472 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" path="/var/lib/kubelet/pods/5367fbb7-b30d-418b-922c-7012d0102dea/volumes" Nov 02 22:00:18.611844 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.611785 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:18.616665 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.616642 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\" (UniqueName: \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\") pod \"f1c6d2df-db0c-4006-b15e-31dda383a373\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " Nov 02 22:00:18.616787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.616729 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\") pod \"f1c6d2df-db0c-4006-b15e-31dda383a373\" (UID: \"f1c6d2df-db0c-4006-b15e-31dda383a373\") " Nov 02 22:00:18.617182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.617158 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca" pod "f1c6d2df-db0c-4006-b15e-31dda383a373" (UID: "f1c6d2df-db0c-4006-b15e-31dda383a373"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:18.618627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.618609 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v" pod "f1c6d2df-db0c-4006-b15e-31dda383a373" (UID: "f1c6d2df-db0c-4006-b15e-31dda383a373"). InnerVolumeSpecName "kube-api-access-nmk4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:18.718245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.718214 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmk4v\" (UniqueName: \"kubernetes.io/projected/f1c6d2df-db0c-4006-b15e-31dda383a373-kube-api-access-nmk4v\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:18.718245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:18.718245 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6d2df-db0c-4006-b15e-31dda383a373-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.348695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.348672 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" Nov 02 22:00:19.349139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.348671 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf" event={"ID":"f1c6d2df-db0c-4006-b15e-31dda383a373","Type":"ContainerDied","Data":"a96053af78daf48adb97352b2f302fdfa78c535b74056ece0dcef322e11a5e67"} Nov 02 22:00:19.349139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.348799 5859 scope.go:122] "RemoveContainer" containerID="b0186b0a7df89fde4c0253d9fc2bdf6e5d5c35ca345a543b71a37ece575c9340" Nov 02 22:00:19.501817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.501793 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:19.571710 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.571684 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:19.576527 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.576500 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894280-h7xvf"] Nov 02 22:00:19.630130 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.630063 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\" (UniqueName: \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\") pod \"65672911-c370-457c-b393-c4d341613524\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " Nov 02 22:00:19.630242 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.630154 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\") pod \"65672911-c370-457c-b393-c4d341613524\" (UID: \"65672911-c370-457c-b393-c4d341613524\") " Nov 02 22:00:19.630472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.630451 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca" pod "65672911-c370-457c-b393-c4d341613524" (UID: "65672911-c370-457c-b393-c4d341613524"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:19.632031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.632010 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk" pod "65672911-c370-457c-b393-c4d341613524" (UID: "65672911-c370-457c-b393-c4d341613524"). InnerVolumeSpecName "kube-api-access-79hwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:19.732197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.732166 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79hwk\" (UniqueName: \"kubernetes.io/projected/65672911-c370-457c-b393-c4d341613524-kube-api-access-79hwk\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:19.732197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:19.732194 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65672911-c370-457c-b393-c4d341613524-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:20.356252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.356217 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" event={"ID":"65672911-c370-457c-b393-c4d341613524","Type":"ContainerDied","Data":"607c388e6c125b37af995a8d4207f1a037ef25473cda93f247c14782f67645fe"} Nov 02 22:00:20.356692 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.356229 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479" Nov 02 22:00:20.356692 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.356289 5859 scope.go:122] "RemoveContainer" containerID="92c1bc9ffc2991c95d1760fa596c0b44448fe1d0b7cdba1dbdc277354d5cf185" Nov 02 22:00:20.520927 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.520892 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:20.526802 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.526777 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894280-k9479"] Nov 02 22:00:20.570631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.570601 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65672911-c370-457c-b393-c4d341613524" path="/var/lib/kubelet/pods/65672911-c370-457c-b393-c4d341613524/volumes" Nov 02 22:00:20.571047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:20.571033 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" path="/var/lib/kubelet/pods/f1c6d2df-db0c-4006-b15e-31dda383a373/volumes" Nov 02 22:00:25.943228 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:25.943190 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:00:30.902607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:30.902568 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 22:00:30.903168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:30.902767 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" containerID="cri-o://f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc" gracePeriod=30 Nov 02 22:00:32.029465 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029393 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:00:32.029981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029935 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.029981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029961 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.029981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029970 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.029981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029975 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.029981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029984 5859 cpu_manager.go:414] "containerMap: removing container" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029990 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.029999 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030004 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030019 5859 cpu_manager.go:414] "containerMap: removing container" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030023 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030052 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030057 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030155 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030165 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030171 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030177 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.030183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030183 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:32.030524 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.030190 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:32.035070 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.035050 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.056492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.056447 5859 patch_prober.go:29] interesting pod/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:00:32.056846 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.056795 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:00:32.073382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.073346 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.073696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.073673 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.073882 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.073854 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.074143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074114 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.074320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074297 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.074394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074359 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.077969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074462 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.077969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074510 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drn2\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.077969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.074640 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.092006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.091054 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4f0c8e-openshift-default-dockercfg-4nfw8\"" Nov 02 22:00:32.146208 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.142129 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:00:32.177364 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177328 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177394 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177445 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177487 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177520 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177576 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177626 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177761 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177794 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.177817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.177806 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drn2\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.178194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.178157 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.178410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.178391 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.179089 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.179065 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.179696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.179322 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.181836 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.181789 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.182530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.182492 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.253838 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.253809 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.254007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.253884 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drn2\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\") pod \"e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.355919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.355895 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:32.832282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.832245 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:00:32.841989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.841920 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:32.842112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.842036 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:32.842112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:32.842089 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:00:33.438434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:33.438399 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" event={"ID":"b5923d92-890f-41d6-827e-2d9929663873","Type":"ContainerStarted","Data":"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d"} Nov 02 22:00:33.438828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:33.438459 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" event={"ID":"b5923d92-890f-41d6-827e-2d9929663873","Type":"ContainerStarted","Data":"86555e3e28bdf2e76e482daccc3813cff7e33fa2b28bff56fa52ffc5e478ef66"} Nov 02 22:00:33.555716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:33.555667 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" podStartSLOduration=2.5556535030000003 podStartE2EDuration="2.555653503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:33.554496495 +0000 UTC m=+3073.102746275" watchObservedRunningTime="2026-11-02 22:00:33.555653503 +0000 UTC m=+3073.103903282" Nov 02 22:00:34.356078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:34.356032 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:34.362514 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:34.362490 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:34.444638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:34.444607 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:34.445513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:34.445497 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:00:36.161822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.161800 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 22:00:36.226947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.226895 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.226947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.226929 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.226997 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227079 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227104 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227121 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227205 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.227371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227240 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.227371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227250 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227276 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.227371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227315 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227646 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227406 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\") pod \"616867c4-1f91-4c94-a67a-61a52d9ee822\" (UID: \"616867c4-1f91-4c94-a67a-61a52d9ee822\") " Nov 02 22:00:36.227646 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227508 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:36.227755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227702 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.228006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.227984 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.228140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.228010 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616867c4-1f91-4c94-a67a-61a52d9ee822-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.228140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.228025 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.228140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.228038 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.228140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.228052 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.229302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.229278 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:36.229302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.229288 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:00:36.229451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.229371 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.229451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.229409 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv" pod "616867c4-1f91-4c94-a67a-61a52d9ee822" (UID: "616867c4-1f91-4c94-a67a-61a52d9ee822"). InnerVolumeSpecName "kube-api-access-55hcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:36.326279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.326253 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 22:00:36.326449 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.326418 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerName="discovery" containerID="cri-o://60e2639cceded820b180d8c76b0cbe0fa68eb10b5491307b9051eb0dfd268c73" gracePeriod=30 Nov 02 22:00:36.330618 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.330552 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hcv\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-kube-api-access-55hcv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.330781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.330586 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.330887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.330791 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.330887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.330805 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616867c4-1f91-4c94-a67a-61a52d9ee822-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.458764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.458734 5859 generic.go:356] "Generic (PLEG): container finished" podID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerID="f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc" exitCode=0 Nov 02 22:00:36.458920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.458798 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" Nov 02 22:00:36.458920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.458814 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerDied","Data":"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc"} Nov 02 22:00:36.458920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.458874 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs" event={"ID":"616867c4-1f91-4c94-a67a-61a52d9ee822","Type":"ContainerDied","Data":"f4d19c4546b413fca5ead0e083260d6b2543e96448d4572cb384337aee4fa3d8"} Nov 02 22:00:36.458920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.458906 5859 scope.go:122] "RemoveContainer" containerID="f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc" Nov 02 22:00:36.460717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.460691 5859 generic.go:356] "Generic (PLEG): container finished" podID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerID="60e2639cceded820b180d8c76b0cbe0fa68eb10b5491307b9051eb0dfd268c73" exitCode=0 Nov 02 22:00:36.460854 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.460766 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" event={"ID":"367d2f6d-011f-4056-a9a4-41849d07e76c","Type":"ContainerDied","Data":"60e2639cceded820b180d8c76b0cbe0fa68eb10b5491307b9051eb0dfd268c73"} Nov 02 22:00:36.469773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.469757 5859 scope.go:122] "RemoveContainer" containerID="b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659" Nov 02 22:00:36.479285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.479260 5859 scope.go:122] "RemoveContainer" containerID="f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc" Nov 02 22:00:36.479537 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:36.479517 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc\": container with ID starting with f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc not found: ID does not exist" containerID="f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc" Nov 02 22:00:36.479612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.479544 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc"} err="failed to get container status \"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc\": rpc error: code = NotFound desc = could not find container \"f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc\": container with ID starting with f0e04d5454a74353490fed2029ec6306d510a1dc516d651e7559d5e474c1bfbc not found: ID does not exist" Nov 02 22:00:36.479612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.479560 5859 scope.go:122] "RemoveContainer" containerID="b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659" Nov 02 22:00:36.479773 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:36.479756 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659\": container with ID starting with b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659 not found: ID does not exist" containerID="b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659" Nov 02 22:00:36.479843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.479776 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659"} err="failed to get container status \"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659\": rpc error: code = NotFound desc = could not find container \"b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659\": container with ID starting with b098ddd17325454c97dea151bf0a0111ea6de34ac49d4ceba19d6d4be0334659 not found: ID does not exist" Nov 02 22:00:36.567190 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.567166 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 22:00:36.627933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.627902 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 22:00:36.631215 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.631194 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4ca09f-openshift-default-595654cb85-fw6cs"] Nov 02 22:00:36.635241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635222 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635257 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635334 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635297 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635334 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635319 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635355 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635468 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635452 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.635575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.635558 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\") pod \"367d2f6d-011f-4056-a9a4-41849d07e76c\" (UID: \"367d2f6d-011f-4056-a9a4-41849d07e76c\") " Nov 02 22:00:36.636660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.636025 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:36.636660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.636236 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-ca-configmap\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.637731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.637703 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:36.637731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.637722 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:36.637987 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.637941 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:00:36.638056 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.638039 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "kube-api-access-qzwgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:36.638119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.638058 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:36.638119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.638099 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token" pod "367d2f6d-011f-4056-a9a4-41849d07e76c" (UID: "367d2f6d-011f-4056-a9a4-41849d07e76c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:36.737506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737459 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwgz\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-kube-api-access-qzwgz\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.737506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737479 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.737506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737488 5859 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/367d2f6d-011f-4056-a9a4-41849d07e76c-local-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.737506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737497 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-csr-dns-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.737506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737505 5859 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-cacerts\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:36.737716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:36.737513 5859 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/367d2f6d-011f-4056-a9a4-41849d07e76c-istio-kubeconfig\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:37.471223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:37.471193 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" Nov 02 22:00:37.471632 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:37.471192 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q" event={"ID":"367d2f6d-011f-4056-a9a4-41849d07e76c","Type":"ContainerDied","Data":"35f8626dddc1b45de47b0158284b04194eec014a52dcc538113388912f5b65c8"} Nov 02 22:00:37.471632 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:37.471339 5859 scope.go:122] "RemoveContainer" containerID="60e2639cceded820b180d8c76b0cbe0fa68eb10b5491307b9051eb0dfd268c73" Nov 02 22:00:37.645205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:37.645179 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 22:00:37.695459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:37.695435 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c882q"] Nov 02 22:00:38.572908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:38.572869 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" path="/var/lib/kubelet/pods/367d2f6d-011f-4056-a9a4-41849d07e76c/volumes" Nov 02 22:00:38.573531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:38.573510 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" path="/var/lib/kubelet/pods/616867c4-1f91-4c94-a67a-61a52d9ee822/volumes" Nov 02 22:00:55.945837 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:00:55.945798 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:00:57.825795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.825756 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:00:57.826424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826407 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:57.826472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826426 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:57.826472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826443 5859 cpu_manager.go:414] "containerMap: removing container" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:57.826472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826448 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:57.826472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826460 5859 cpu_manager.go:414] "containerMap: removing container" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerName="discovery" Nov 02 22:00:57.826472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826465 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerName="discovery" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826475 5859 cpu_manager.go:414] "containerMap: removing container" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826479 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826492 5859 cpu_manager.go:414] "containerMap: removing container" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826496 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826526 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826532 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:57.826623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826615 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c6d2df-db0c-4006-b15e-31dda383a373" containerName="cleanup" Nov 02 22:00:57.826829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826626 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5367fbb7-b30d-418b-922c-7012d0102dea" containerName="cleanup" Nov 02 22:00:57.826829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826635 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="65672911-c370-457c-b393-c4d341613524" containerName="cleanup" Nov 02 22:00:57.826829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826640 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="367d2f6d-011f-4056-a9a4-41849d07e76c" containerName="discovery" Nov 02 22:00:57.826829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826645 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.826829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.826652 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="616867c4-1f91-4c94-a67a-61a52d9ee822" containerName="istio-proxy" Nov 02 22:00:57.830119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.830102 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.871656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.871620 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.871861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.871681 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.871861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.871810 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.872021 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.871872 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.872021 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.871932 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb84g\" (UniqueName: \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.872152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.872038 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.885555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.885515 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:00:57.931447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.931410 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"openshift-service-ca.crt\"" Nov 02 22:00:57.931447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.931444 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"kube-root-ca.crt\"" Nov 02 22:00:57.931447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.931440 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"default-dockercfg-ptbhj\"" Nov 02 22:00:57.932911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.932892 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"auth-test-model-58ca7f-kserve-self-signed-certs\"" Nov 02 22:00:57.973366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973323 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973386 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb84g\" (UniqueName: \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973407 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973490 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973515 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973661 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.973981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973923 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.974076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.973935 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.974137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.974091 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.976013 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.975993 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:57.976138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:57.976015 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:58.030723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.030690 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb84g\" (UniqueName: \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\") pod \"auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:58.143180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.143090 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:00:58.516738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.516712 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:00:58.518685 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:00:58.518659 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4711eddf_5a4e_4fe1_91d1_9b88ce6bbde3.slice/crio-79f6405c2cdb2c3a4e52e10fa6c3d807c6a5e8489c72291e601c1ddc4d6a5702 WatchSource:0}: Error finding container 79f6405c2cdb2c3a4e52e10fa6c3d807c6a5e8489c72291e601c1ddc4d6a5702: Status 404 returned error can't find the container with id 79f6405c2cdb2c3a4e52e10fa6c3d807c6a5e8489c72291e601c1ddc4d6a5702 Nov 02 22:00:58.605517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.605480 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" event={"ID":"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3","Type":"ContainerStarted","Data":"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2"} Nov 02 22:00:58.605703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.605530 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" event={"ID":"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3","Type":"ContainerStarted","Data":"79f6405c2cdb2c3a4e52e10fa6c3d807c6a5e8489c72291e601c1ddc4d6a5702"} Nov 02 22:00:58.721219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:58.721158 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" podStartSLOduration=1.721144767 podStartE2EDuration="1.721144767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:58.720535133 +0000 UTC m=+3098.268784932" watchObservedRunningTime="2026-11-02 22:00:58.721144767 +0000 UTC m=+3098.269394547" Nov 02 22:00:59.143184 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:00:59.143146 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:01:05.066113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.066075 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 22:01:05.066535 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.066362 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" containerID="cri-o://48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa" gracePeriod=30 Nov 02 22:01:05.579717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.579685 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:01:05.583789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.583767 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.636724 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.636693 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4f0c8e\"/\"openshift-service-ca.crt\"" Nov 02 22:01:05.638711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.638684 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:01:05.667186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.667155 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4f0c8e\"/\"kube-root-ca.crt\"" Nov 02 22:01:05.667186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.667169 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4f0c8e\"/\"default-dockercfg-vkcvc\"" Nov 02 22:01:05.672816 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.672786 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.672964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.672935 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.673128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.673104 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.673204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.673154 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.673433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.673413 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746kx\" (UniqueName: \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.673469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.673457 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.685638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.685623 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4f0c8e\"/\"noten-model-4f0c8e-kserve-self-signed-certs\"" Nov 02 22:01:05.774855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.774820 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.774912 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.774937 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775066 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746kx\" (UniqueName: \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775100 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775165 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775255 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775520 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775498 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.775653 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.775501 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.777253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.777237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.777574 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.777556 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.831722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.831655 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746kx\" (UniqueName: \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\") pod \"noten-model-4f0c8e-kserve-844c9d8f7-f45sj\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:05.897183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:05.897154 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:06.275221 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:06.275197 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:01:06.276089 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:01:06.276063 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd12a26fa_06dc_4ba4_aa1f_d96eae27f784.slice/crio-56ec691a769852f2034b298d8caca24170a303e2cfaf7fb55222f57879b1108c WatchSource:0}: Error finding container 56ec691a769852f2034b298d8caca24170a303e2cfaf7fb55222f57879b1108c: Status 404 returned error can't find the container with id 56ec691a769852f2034b298d8caca24170a303e2cfaf7fb55222f57879b1108c Nov 02 22:01:06.657750 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:06.657712 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" event={"ID":"d12a26fa-06dc-4ba4-aa1f-d96eae27f784","Type":"ContainerStarted","Data":"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69"} Nov 02 22:01:06.657750 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:06.657756 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" event={"ID":"d12a26fa-06dc-4ba4-aa1f-d96eae27f784","Type":"ContainerStarted","Data":"56ec691a769852f2034b298d8caca24170a303e2cfaf7fb55222f57879b1108c"} Nov 02 22:01:06.771512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:06.771464 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" podStartSLOduration=1.771450207 podStartE2EDuration="1.771450207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:06.770636909 +0000 UTC m=+3106.318886692" watchObservedRunningTime="2026-11-02 22:01:06.771450207 +0000 UTC m=+3106.319699988" Nov 02 22:01:06.897641 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:06.897606 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:08.143658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:08.143629 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:01:08.155910 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:08.155886 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:01:08.683085 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:08.683055 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:01:09.475845 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:09.475806 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:09.476254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:09.475865 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:10.320541 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.320513 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 22:01:10.438887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.438822 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.438887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.438879 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.438904 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439013 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439051 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439104 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439135 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439172 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439239 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\") pod \"6da78730-8bdf-4110-9e91-ce41d18448c6\" (UID: \"6da78730-8bdf-4110-9e91-ce41d18448c6\") " Nov 02 22:01:10.439340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439235 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.439549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439464 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.439549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439509 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.439662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439567 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:10.439662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.439633 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.440600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.440573 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.440718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.440603 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.440718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.440616 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da78730-8bdf-4110-9e91-ce41d18448c6-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.440718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.440624 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.440718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.440632 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.441773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.441746 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:10.441929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.441904 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.442252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.442225 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:10.442252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.442234 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2" pod "6da78730-8bdf-4110-9e91-ce41d18448c6" (UID: "6da78730-8bdf-4110-9e91-ce41d18448c6"). InnerVolumeSpecName "kube-api-access-5xhl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:10.541800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.541766 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.541800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.541798 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-kube-api-access-5xhl2\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.541800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.541808 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.542259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.541818 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da78730-8bdf-4110-9e91-ce41d18448c6-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.685971 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.685924 5859 generic.go:356] "Generic (PLEG): container finished" podID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerID="48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa" exitCode=0 Nov 02 22:01:10.686161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.685982 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" event={"ID":"6da78730-8bdf-4110-9e91-ce41d18448c6","Type":"ContainerDied","Data":"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa"} Nov 02 22:01:10.686161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.686009 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" Nov 02 22:01:10.686161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.686038 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv" event={"ID":"6da78730-8bdf-4110-9e91-ce41d18448c6","Type":"ContainerDied","Data":"c503a25fc012543bd8eb1d01f36961526b07de3c706e9311a648e15c1518edec"} Nov 02 22:01:10.686161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.686065 5859 scope.go:122] "RemoveContainer" containerID="48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa" Nov 02 22:01:10.696197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.696181 5859 scope.go:122] "RemoveContainer" containerID="48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa" Nov 02 22:01:10.696430 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:01:10.696414 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa\": container with ID starting with 48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa not found: ID does not exist" containerID="48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa" Nov 02 22:01:10.696477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.696440 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa"} err="failed to get container status \"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa\": rpc error: code = NotFound desc = could not find container \"48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa\": container with ID starting with 48fc0d4ecfeaaf537e97e53421eeb174c3e46f65c42b944734f4bf6d8709f9fa not found: ID does not exist" Nov 02 22:01:10.852846 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.852812 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 22:01:10.856608 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:10.856579 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-8b8cv"] Nov 02 22:01:12.571651 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:12.571617 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" path="/var/lib/kubelet/pods/6da78730-8bdf-4110-9e91-ce41d18448c6/volumes" Nov 02 22:01:15.897633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:15.897592 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:15.910253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:15.910218 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:16.736598 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:16.736564 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:01:25.943636 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:01:25.943595 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:01:47.123939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:47.123905 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:01:47.124534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:47.124164 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" podUID="b5923d92-890f-41d6-827e-2d9929663873" containerName="istio-proxy" containerID="cri-o://17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d" gracePeriod=30 Nov 02 22:01:49.440630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.440595 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:01:49.441457 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.441436 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" Nov 02 22:01:49.441513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.441461 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" Nov 02 22:01:49.441667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.441654 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da78730-8bdf-4110-9e91-ce41d18448c6" containerName="istio-proxy" Nov 02 22:01:49.445857 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.445826 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.447392 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.447365 5859 patch_prober.go:29] interesting pod/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:49.447519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.447444 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" podUID="b5923d92-890f-41d6-827e-2d9929663873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:49.482603 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482578 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.482749 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482644 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.482801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482757 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwsc\" (UniqueName: \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.482847 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482831 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.482888 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482864 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.482995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.482939 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.498717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.498688 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:01:49.544789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.544747 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-06e461\"/\"default-dockercfg-vkqkp\"" Nov 02 22:01:49.544962 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.544822 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-06e461\"/\"auth-test-model-06e461-kserve-self-signed-certs\"" Nov 02 22:01:49.548721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.548697 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-06e461\"/\"kube-root-ca.crt\"" Nov 02 22:01:49.548849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.548737 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-06e461\"/\"openshift-service-ca.crt\"" Nov 02 22:01:49.583859 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.583823 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.583915 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584010 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584059 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwsc\" (UniqueName: \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584131 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584162 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584764 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.584980 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.584883 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.585325 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.585184 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.587871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.587846 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.588394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.588371 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.641700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.641670 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwsc\" (UniqueName: \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\") pod \"auth-test-model-06e461-kserve-8468c86b45-vn4qt\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:49.759537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:49.759465 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:50.142975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:50.142930 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:01:50.144168 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:01:50.144140 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3719044_505f_418b_8dbd_c9686a9e2c0a.slice/crio-2471aa67b08589bdabf59a01112cb2378ae76695f2d3e412a2c559a6653ef865 WatchSource:0}: Error finding container 2471aa67b08589bdabf59a01112cb2378ae76695f2d3e412a2c559a6653ef865: Status 404 returned error can't find the container with id 2471aa67b08589bdabf59a01112cb2378ae76695f2d3e412a2c559a6653ef865 Nov 02 22:01:50.942849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:50.942815 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" event={"ID":"c3719044-505f-418b-8dbd-c9686a9e2c0a","Type":"ContainerStarted","Data":"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82"} Nov 02 22:01:50.943248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:50.942859 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" event={"ID":"c3719044-505f-418b-8dbd-c9686a9e2c0a","Type":"ContainerStarted","Data":"2471aa67b08589bdabf59a01112cb2378ae76695f2d3e412a2c559a6653ef865"} Nov 02 22:01:51.058833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:51.058777 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" podStartSLOduration=2.058764164 podStartE2EDuration="2.058764164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:51.056433971 +0000 UTC m=+3150.604683770" watchObservedRunningTime="2026-11-02 22:01:51.058764164 +0000 UTC m=+3150.607013944" Nov 02 22:01:51.759551 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:51.759516 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:52.382424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.382365 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:01:52.415745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415721 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.415885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415767 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.415885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415788 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.415885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415834 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416070 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415918 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416070 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.415944 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416070 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416052 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416141 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416170 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\") pod \"b5923d92-890f-41d6-827e-2d9929663873\" (UID: \"b5923d92-890f-41d6-827e-2d9929663873\") " Nov 02 22:01:52.416232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416048 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:52.416232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416169 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:52.416451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416208 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:52.416451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416276 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:52.416451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416424 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:52.416714 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416694 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.416813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416715 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5923d92-890f-41d6-827e-2d9929663873-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.416813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416730 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.416813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416738 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.416813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.416748 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.418269 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.418249 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:52.418371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.418329 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "kube-api-access-8drn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:52.418615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.418599 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:52.418657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.418626 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy" pod "b5923d92-890f-41d6-827e-2d9929663873" (UID: "b5923d92-890f-41d6-827e-2d9929663873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:52.518152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.518130 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5923d92-890f-41d6-827e-2d9929663873-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.518152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.518153 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8drn2\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-kube-api-access-8drn2\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.518281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.518162 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5923d92-890f-41d6-827e-2d9929663873-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.518281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.518172 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5923d92-890f-41d6-827e-2d9929663873-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:52.605578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.605551 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:01:52.956105 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.956068 5859 generic.go:356] "Generic (PLEG): container finished" podID="b5923d92-890f-41d6-827e-2d9929663873" containerID="17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d" exitCode=0 Nov 02 22:01:52.956331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.956113 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" Nov 02 22:01:52.956331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.956149 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" event={"ID":"b5923d92-890f-41d6-827e-2d9929663873","Type":"ContainerDied","Data":"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d"} Nov 02 22:01:52.956331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.956198 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9" event={"ID":"b5923d92-890f-41d6-827e-2d9929663873","Type":"ContainerDied","Data":"86555e3e28bdf2e76e482daccc3813cff7e33fa2b28bff56fa52ffc5e478ef66"} Nov 02 22:01:52.956331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.956221 5859 scope.go:122] "RemoveContainer" containerID="17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d" Nov 02 22:01:52.966059 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.966041 5859 scope.go:122] "RemoveContainer" containerID="17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d" Nov 02 22:01:52.966336 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:01:52.966318 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d\": container with ID starting with 17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d not found: ID does not exist" containerID="17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d" Nov 02 22:01:52.966416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:52.966341 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d"} err="failed to get container status \"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d\": rpc error: code = NotFound desc = could not find container \"17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d\": container with ID starting with 17ce2422eb3437baf96c61270d64e0a5ed4fad9c9d6b1b19434e48191d133a6d not found: ID does not exist" Nov 02 22:01:53.120828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:53.120801 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:01:53.125607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:53.125587 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4f0c8e-openshift-default-54774cc65c-84xb9"] Nov 02 22:01:54.571471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:54.571432 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5923d92-890f-41d6-827e-2d9929663873" path="/var/lib/kubelet/pods/b5923d92-890f-41d6-827e-2d9929663873/volumes" Nov 02 22:01:55.945468 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:01:55.945435 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:01:59.760357 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:59.760330 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:01:59.773072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:01:59.773048 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:02:00.012667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:00.012609 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:02:07.625710 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.625620 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerName="main" containerID="cri-o://baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69" gracePeriod=45 Nov 02 22:02:07.726790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.726757 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.151:8000/ready\": dial tcp 10.134.0.151:8000: connect: connection refused" Nov 02 22:02:07.882873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.882820 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:02:07.973886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.973857 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\" (UniqueName: \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.973936 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974023 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974166 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974061 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974166 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974139 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\" (UniqueName: \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974166 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974139 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:07.974320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974229 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\") pod \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\" (UID: \"d12a26fa-06dc-4ba4-aa1f-d96eae27f784\") " Nov 02 22:02:07.974320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974261 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:07.974518 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974497 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:07.974723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974707 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:07.974723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974724 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:07.974863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.974733 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:07.976186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.976161 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:07.976186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.976162 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "kube-api-access-746kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:07.976309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:07.976233 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm" pod "d12a26fa-06dc-4ba4-aa1f-d96eae27f784" (UID: "d12a26fa-06dc-4ba4-aa1f-d96eae27f784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:08.051561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.051530 5859 generic.go:356] "Generic (PLEG): container finished" podID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerID="baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69" exitCode=0 Nov 02 22:02:08.051689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.051573 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" Nov 02 22:02:08.051689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.051614 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" event={"ID":"d12a26fa-06dc-4ba4-aa1f-d96eae27f784","Type":"ContainerDied","Data":"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69"} Nov 02 22:02:08.051689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.051657 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj" event={"ID":"d12a26fa-06dc-4ba4-aa1f-d96eae27f784","Type":"ContainerDied","Data":"56ec691a769852f2034b298d8caca24170a303e2cfaf7fb55222f57879b1108c"} Nov 02 22:02:08.051689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.051677 5859 scope.go:122] "RemoveContainer" containerID="baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69" Nov 02 22:02:08.062410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.062394 5859 scope.go:122] "RemoveContainer" containerID="baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69" Nov 02 22:02:08.062653 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:08.062638 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69\": container with ID starting with baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69 not found: ID does not exist" containerID="baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69" Nov 02 22:02:08.062697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.062661 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69"} err="failed to get container status \"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69\": rpc error: code = NotFound desc = could not find container \"baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69\": container with ID starting with baeabb4f17346ad6e025159594e7e3f6e0b037a90d45320cc189704411dcfd69 not found: ID does not exist" Nov 02 22:02:08.076354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.076336 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746kx\" (UniqueName: \"kubernetes.io/projected/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-kube-api-access-746kx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.076423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.076356 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.076423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.076365 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12a26fa-06dc-4ba4-aa1f-d96eae27f784-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.216179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.216112 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:02:08.218551 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.218530 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4f0c8e/noten-model-4f0c8e-kserve-844c9d8f7-f45sj"] Nov 02 22:02:08.571261 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:08.571233 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" path="/var/lib/kubelet/pods/d12a26fa-06dc-4ba4-aa1f-d96eae27f784/volumes" Nov 02 22:02:25.944052 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:25.944008 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:02:29.862921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.862890 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7471-c492df"] Nov 02 22:02:29.863435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863422 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b5923d92-890f-41d6-827e-2d9929663873" containerName="istio-proxy" Nov 02 22:02:29.863435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863435 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5923d92-890f-41d6-827e-2d9929663873" containerName="istio-proxy" Nov 02 22:02:29.863509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863461 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerName="main" Nov 02 22:02:29.863509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863466 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerName="main" Nov 02 22:02:29.863579 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863570 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5923d92-890f-41d6-827e-2d9929663873" containerName="istio-proxy" Nov 02 22:02:29.863614 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.863583 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12a26fa-06dc-4ba4-aa1f-d96eae27f784" containerName="main" Nov 02 22:02:29.867283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.867265 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:29.919573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.919545 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-c492df"] Nov 02 22:02:29.965813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.965790 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\"" Nov 02 22:02:29.991553 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:29.991529 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkw6h\" (UniqueName: \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\") pod \"test-curl-7471-c492df\" (UID: \"e729780f-2305-44f3-b9fd-0a55410c3d58\") " pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:30.092539 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:30.092514 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkw6h\" (UniqueName: \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\") pod \"test-curl-7471-c492df\" (UID: \"e729780f-2305-44f3-b9fd-0a55410c3d58\") " pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:30.149415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:30.149361 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkw6h\" (UniqueName: \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\") pod \"test-curl-7471-c492df\" (UID: \"e729780f-2305-44f3-b9fd-0a55410c3d58\") " pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:30.180195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:30.180172 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:30.551761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:30.551737 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-c492df"] Nov 02 22:02:30.553067 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:02:30.553039 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode729780f_2305_44f3_b9fd_0a55410c3d58.slice/crio-4c3c3df7b992765cadc16375c691660cce1008b378fcf59ebf5adf8495d58e6b WatchSource:0}: Error finding container 4c3c3df7b992765cadc16375c691660cce1008b378fcf59ebf5adf8495d58e6b: Status 404 returned error can't find the container with id 4c3c3df7b992765cadc16375c691660cce1008b378fcf59ebf5adf8495d58e6b Nov 02 22:02:31.197154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:31.197124 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-c492df" event={"ID":"e729780f-2305-44f3-b9fd-0a55410c3d58","Type":"ContainerStarted","Data":"4c3c3df7b992765cadc16375c691660cce1008b378fcf59ebf5adf8495d58e6b"} Nov 02 22:02:32.205932 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:32.205858 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-c492df" event={"ID":"e729780f-2305-44f3-b9fd-0a55410c3d58","Type":"ContainerStarted","Data":"5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223"} Nov 02 22:02:32.318481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:32.318436 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7471-c492df" podStartSLOduration=1.994389513 podStartE2EDuration="3.318421662s" totalImagesPullingTime="1.324032149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:32.317052071 +0000 UTC m=+3191.865301852" watchObservedRunningTime="2026-11-02 22:02:32.318421662 +0000 UTC m=+3191.866671444" Nov 02 22:02:33.658731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.658700 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7471-c492df"] Nov 02 22:02:33.662605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.662576 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7471-c492df"] Nov 02 22:02:33.813735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.813704 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7471-6168b9"] Nov 02 22:02:33.814256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.814242 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" containerName="test-curl-7471-c492df" Nov 02 22:02:33.814256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.814257 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" containerName="test-curl-7471-c492df" Nov 02 22:02:33.814391 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.814383 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" containerName="test-curl-7471-c492df" Nov 02 22:02:33.817993 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.817974 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:33.867239 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.867213 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-6168b9"] Nov 02 22:02:33.932513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:33.932454 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qpj\" (UniqueName: \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\") pod \"test-curl-7471-6168b9\" (UID: \"2b9caa5e-bc92-459e-aa85-4e534706420a\") " pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:34.033302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:34.033274 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qpj\" (UniqueName: \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\") pod \"test-curl-7471-6168b9\" (UID: \"2b9caa5e-bc92-459e-aa85-4e534706420a\") " pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:34.093213 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:34.093191 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qpj\" (UniqueName: \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\") pod \"test-curl-7471-6168b9\" (UID: \"2b9caa5e-bc92-459e-aa85-4e534706420a\") " pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:34.130134 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:34.130112 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:34.220986 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:34.218416 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7471-c492df" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" containerName="test-curl-7471-c492df" containerID="cri-o://5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223" gracePeriod=2 Nov 02 22:02:34.269894 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:34.269857 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-c492df\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:34.501936 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:34.501913 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-6168b9"] Nov 02 22:02:34.503024 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:02:34.503000 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b9caa5e_bc92_459e_aa85_4e534706420a.slice/crio-50a9e56660c908353a3ea3b4b95e36e46807999a8749e7b5b214046ba5a5f8d1 WatchSource:0}: Error finding container 50a9e56660c908353a3ea3b4b95e36e46807999a8749e7b5b214046ba5a5f8d1: Status 404 returned error can't find the container with id 50a9e56660c908353a3ea3b4b95e36e46807999a8749e7b5b214046ba5a5f8d1 Nov 02 22:02:35.226037 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:35.226008 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-6168b9" event={"ID":"2b9caa5e-bc92-459e-aa85-4e534706420a","Type":"ContainerStarted","Data":"d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42"} Nov 02 22:02:35.226493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:35.226050 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-6168b9" event={"ID":"2b9caa5e-bc92-459e-aa85-4e534706420a","Type":"ContainerStarted","Data":"50a9e56660c908353a3ea3b4b95e36e46807999a8749e7b5b214046ba5a5f8d1"} Nov 02 22:02:35.339858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:35.339806 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7471-6168b9" podStartSLOduration=2.339792968 podStartE2EDuration="2.339792968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:35.338201829 +0000 UTC m=+3194.886451610" watchObservedRunningTime="2026-11-02 22:02:35.339792968 +0000 UTC m=+3194.888042748" Nov 02 22:02:36.436427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.436404 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:36.558617 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.558547 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\" (UniqueName: \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\") pod \"e729780f-2305-44f3-b9fd-0a55410c3d58\" (UID: \"e729780f-2305-44f3-b9fd-0a55410c3d58\") " Nov 02 22:02:36.560603 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.560578 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h" pod "e729780f-2305-44f3-b9fd-0a55410c3d58" (UID: "e729780f-2305-44f3-b9fd-0a55410c3d58"). InnerVolumeSpecName "kube-api-access-zkw6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:36.571468 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.571441 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e729780f-2305-44f3-b9fd-0a55410c3d58" path="/var/lib/kubelet/pods/e729780f-2305-44f3-b9fd-0a55410c3d58/volumes" Nov 02 22:02:36.659939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.659910 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkw6h\" (UniqueName: \"kubernetes.io/projected/e729780f-2305-44f3-b9fd-0a55410c3d58-kube-api-access-zkw6h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:36.731096 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.731057 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7471-6168b9"] Nov 02 22:02:36.733537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:36.733512 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7471-6168b9"] Nov 02 22:02:37.239936 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.239896 5859 generic.go:356] "Generic (PLEG): container finished" podID="e729780f-2305-44f3-b9fd-0a55410c3d58" containerID="5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223" exitCode=137 Nov 02 22:02:37.239936 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.239936 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-c492df" Nov 02 22:02:37.240219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.239992 5859 scope.go:122] "RemoveContainer" containerID="5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223" Nov 02 22:02:37.240219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.240171 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7471-6168b9" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" containerName="test-curl-7471-6168b9" containerID="cri-o://d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42" gracePeriod=2 Nov 02 22:02:37.249433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.249402 5859 scope.go:122] "RemoveContainer" containerID="5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223" Nov 02 22:02:37.249710 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:37.249693 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223\": container with ID starting with 5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223 not found: ID does not exist" containerID="5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223" Nov 02 22:02:37.249763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:37.249719 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223"} err="failed to get container status \"5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223\": rpc error: code = NotFound desc = could not find container \"5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223\": container with ID starting with 5c91f60fb69b02b76a52be33e3cb7a8c9fd34cb5c70812fabc9e890d8a5af223 not found: ID does not exist" Nov 02 22:02:37.291523 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:37.291494 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-6168b9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:37.342759 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:37.342735 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-6168b9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:38.069233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.069199 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:02:38.069783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.069770 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" containerName="test-curl-7471-6168b9" Nov 02 22:02:38.069831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.069786 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" containerName="test-curl-7471-6168b9" Nov 02 22:02:38.069923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.069914 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" containerName="test-curl-7471-6168b9" Nov 02 22:02:38.074791 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.074771 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.126238 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:38.126204 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-6168b9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:38.168569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.168536 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:02:38.173526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173497 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173530 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173656 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173694 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173721 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173761 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.173819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173781 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.174051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173889 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6sd\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.174051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.173938 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.215497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.215469 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-6689da-openshift-default-dockercfg-pl6hw\"" Nov 02 22:02:38.275465 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275433 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.275643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275507 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.275643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275534 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.275643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275616 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.275806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275676 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.275806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275701 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276010 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276228 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276427 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.275730 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276493 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276619 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276505 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276636 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6sd\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.276725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.276701 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.277965 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.277929 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.278365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.278344 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.332625 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.332573 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.332753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.332735 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6sd\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\") pod \"e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.391539 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.391514 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:38.823801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.823773 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:02:38.825296 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:02:38.825271 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb724c917_6f74_45a9_b517_bf5d039c678a.slice/crio-399ca3025deb3c8566a7110cdfff34b68648bbbb5141a6215723348373480c75 WatchSource:0}: Error finding container 399ca3025deb3c8566a7110cdfff34b68648bbbb5141a6215723348373480c75: Status 404 returned error can't find the container with id 399ca3025deb3c8566a7110cdfff34b68648bbbb5141a6215723348373480c75 Nov 02 22:02:38.827947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.827909 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:38.828046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.828026 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:38.828091 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:38.828074 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:02:39.256198 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.256167 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerStarted","Data":"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923"} Nov 02 22:02:39.256499 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.256223 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerStarted","Data":"399ca3025deb3c8566a7110cdfff34b68648bbbb5141a6215723348373480c75"} Nov 02 22:02:39.373963 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.373913 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podStartSLOduration=1.373898903 podStartE2EDuration="1.373898903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:39.373078906 +0000 UTC m=+3198.921328706" watchObservedRunningTime="2026-11-02 22:02:39.373898903 +0000 UTC m=+3198.922148683" Nov 02 22:02:39.391803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.391777 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:39.398385 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.398364 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:39.528407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.528384 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:39.594786 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.594760 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\" (UniqueName: \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\") pod \"2b9caa5e-bc92-459e-aa85-4e534706420a\" (UID: \"2b9caa5e-bc92-459e-aa85-4e534706420a\") " Nov 02 22:02:39.596681 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.596658 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj" pod "2b9caa5e-bc92-459e-aa85-4e534706420a" (UID: "2b9caa5e-bc92-459e-aa85-4e534706420a"). InnerVolumeSpecName "kube-api-access-l2qpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:39.696387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:39.696365 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qpj\" (UniqueName: \"kubernetes.io/projected/2b9caa5e-bc92-459e-aa85-4e534706420a-kube-api-access-l2qpj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:40.271531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.271499 5859 generic.go:356] "Generic (PLEG): container finished" podID="2b9caa5e-bc92-459e-aa85-4e534706420a" containerID="d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42" exitCode=137 Nov 02 22:02:40.272063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.271548 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:40.272063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.271599 5859 scope.go:122] "RemoveContainer" containerID="d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42" Nov 02 22:02:40.272063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.271963 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:40.273130 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.273108 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:02:40.281521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.281504 5859 scope.go:122] "RemoveContainer" containerID="d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42" Nov 02 22:02:40.281791 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:40.281770 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42\": container with ID starting with d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42 not found: ID does not exist" containerID="d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42" Nov 02 22:02:40.281843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.281799 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42"} err="failed to get container status \"d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42\": rpc error: code = NotFound desc = could not find container \"d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42\": container with ID starting with d621fdb339f510844c987ad0d3295e416a9c4263b166f07c50585e82d4f3cd42 not found: ID does not exist" Nov 02 22:02:40.439924 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:40.439892 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-6168b9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:40.571351 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:40.571323 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" path="/var/lib/kubelet/pods/2b9caa5e-bc92-459e-aa85-4e534706420a/volumes" Nov 02 22:02:40.618850 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:40.618819 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-6168b9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="2b9caa5e-bc92-459e-aa85-4e534706420a" pod="opendatahub/test-curl-7471-6168b9" Nov 02 22:02:45.751754 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.751718 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:02:45.757530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.757512 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.809780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.809758 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:02:45.857982 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.857372 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6689da\"" Nov 02 22:02:45.857982 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.857534 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-6689da-dockercfg-zcj59\"" Nov 02 22:02:45.863445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.863404 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.863827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.863806 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6zl6\" (UniqueName: \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.924382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.924352 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:02:45.928696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.928676 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:45.965796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.965762 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.965920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.965853 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6zl6\" (UniqueName: \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.966443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.966424 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:45.982136 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:45.982108 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:02:46.023572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.023520 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6zl6\" (UniqueName: \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\") pod \"payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:46.066977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.066925 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkqh\" (UniqueName: \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.067131 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.067107 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.070905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.070889 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:46.168398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.168368 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.168548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.168535 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkqh\" (UniqueName: \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.169336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.169312 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.225835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.225804 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkqh\" (UniqueName: \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\") pod \"payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.242463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.242443 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:46.457150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.457126 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:02:46.458647 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:02:46.458621 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c160b12_f67c_46fc_83a9_8ff5ce482ff9.slice/crio-df65e80fd1b3533abad300ed3cc94e67d8925aeb0e9a7cb80565f87293d2f136 WatchSource:0}: Error finding container df65e80fd1b3533abad300ed3cc94e67d8925aeb0e9a7cb80565f87293d2f136: Status 404 returned error can't find the container with id df65e80fd1b3533abad300ed3cc94e67d8925aeb0e9a7cb80565f87293d2f136 Nov 02 22:02:46.614363 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:46.614340 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:02:46.615325 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:02:46.615299 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeac2e05_049b_457a_ad96_ab5b83e33d62.slice/crio-a1184927b569b70809a0fc5c06922a3eade25c531bc77d9cf0fe6b7a0948f5d4 WatchSource:0}: Error finding container a1184927b569b70809a0fc5c06922a3eade25c531bc77d9cf0fe6b7a0948f5d4: Status 404 returned error can't find the container with id a1184927b569b70809a0fc5c06922a3eade25c531bc77d9cf0fe6b7a0948f5d4 Nov 02 22:02:47.321793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.321759 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" event={"ID":"1c160b12-f67c-46fc-83a9-8ff5ce482ff9","Type":"ContainerStarted","Data":"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748"} Nov 02 22:02:47.322217 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.321812 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" event={"ID":"1c160b12-f67c-46fc-83a9-8ff5ce482ff9","Type":"ContainerStarted","Data":"df65e80fd1b3533abad300ed3cc94e67d8925aeb0e9a7cb80565f87293d2f136"} Nov 02 22:02:47.322217 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.321849 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:47.323556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.323533 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" event={"ID":"aeac2e05-049b-457a-ad96-ab5b83e33d62","Type":"ContainerStarted","Data":"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c"} Nov 02 22:02:47.323673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.323570 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" event={"ID":"aeac2e05-049b-457a-ad96-ab5b83e33d62","Type":"ContainerStarted","Data":"a1184927b569b70809a0fc5c06922a3eade25c531bc77d9cf0fe6b7a0948f5d4"} Nov 02 22:02:47.323673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.323608 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:02:47.443633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.443584 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" podStartSLOduration=2.443571597 podStartE2EDuration="2.443571597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:47.441199965 +0000 UTC m=+3206.989449764" watchObservedRunningTime="2026-11-02 22:02:47.443571597 +0000 UTC m=+3206.991821376" Nov 02 22:02:47.556721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:47.556672 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" podStartSLOduration=2.556653478 podStartE2EDuration="2.556653478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:02:47.555434621 +0000 UTC m=+3207.103684416" watchObservedRunningTime="2026-11-02 22:02:47.556653478 +0000 UTC m=+3207.104903258" Nov 02 22:02:55.944407 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:02:55.944370 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:02:58.333291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:58.333261 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:02:58.333697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:02:58.333368 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:03:15.428244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.428216 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:03:15.432159 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.432143 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.488340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.488314 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:03:15.534875 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.534840 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c7e71f\"/\"kube-root-ca.crt\"" Nov 02 22:03:15.535041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.534885 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c7e71f\"/\"openshift-service-ca.crt\"" Nov 02 22:03:15.535041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.534916 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c7e71f\"/\"default-dockercfg-ldvb5\"" Nov 02 22:03:15.535222 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.535208 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c7e71f\"/\"auth-test-model-c7e71f-kserve-self-signed-certs\"" Nov 02 22:03:15.555203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555174 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.555307 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555233 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.555364 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555332 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.555403 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555381 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.555464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555451 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.555532 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.555519 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47vg\" (UniqueName: \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656449 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656510 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47vg\" (UniqueName: \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656605 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656733 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656665 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656737 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656781 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.656911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.656794 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.657030 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.657008 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.657101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.657058 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.658927 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.658897 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.659168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.659150 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.713818 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.713763 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47vg\" (UniqueName: \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\") pod \"auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:15.745638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:15.745617 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:16.119108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:16.119081 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:03:16.120882 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:16.120853 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74b1ce7c_f811_45ac_80be_38958d3386d3.slice/crio-c5fafb63a3888bfca4851366c7c89feb13c31d272965993f3d674ac382f5fdc9 WatchSource:0}: Error finding container c5fafb63a3888bfca4851366c7c89feb13c31d272965993f3d674ac382f5fdc9: Status 404 returned error can't find the container with id c5fafb63a3888bfca4851366c7c89feb13c31d272965993f3d674ac382f5fdc9 Nov 02 22:03:16.511094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:16.511013 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" event={"ID":"74b1ce7c-f811-45ac-80be-38958d3386d3","Type":"ContainerStarted","Data":"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1"} Nov 02 22:03:16.511094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:16.511059 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" event={"ID":"74b1ce7c-f811-45ac-80be-38958d3386d3","Type":"ContainerStarted","Data":"c5fafb63a3888bfca4851366c7c89feb13c31d272965993f3d674ac382f5fdc9"} Nov 02 22:03:16.629320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:16.629275 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" podStartSLOduration=1.629262571 podStartE2EDuration="1.629262571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:16.62670966 +0000 UTC m=+3236.174959462" watchObservedRunningTime="2026-11-02 22:03:16.629262571 +0000 UTC m=+3236.177512342" Nov 02 22:03:16.746116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:16.746086 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:25.746464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:25.746427 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:25.759580 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:25.759555 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:25.944459 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:25.944424 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:03:26.584075 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:26.584046 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:03:27.241795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.241760 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-21c31f"] Nov 02 22:03:27.245783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.245766 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:27.295876 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.295847 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-21c31f"] Nov 02 22:03:27.308250 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.308217 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twd4f\" (UniqueName: \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\") pod \"test-curl-7477-21c31f\" (UID: \"b0640402-3935-40dc-aa96-8a93b76cd9a4\") " pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:27.342650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.342615 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\"" Nov 02 22:03:27.409494 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.409458 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twd4f\" (UniqueName: \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\") pod \"test-curl-7477-21c31f\" (UID: \"b0640402-3935-40dc-aa96-8a93b76cd9a4\") " pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:27.466400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.466373 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twd4f\" (UniqueName: \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\") pod \"test-curl-7477-21c31f\" (UID: \"b0640402-3935-40dc-aa96-8a93b76cd9a4\") " pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:27.559128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.559047 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:27.929431 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:27.929402 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-21c31f"] Nov 02 22:03:27.931774 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:27.931752 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0640402_3935_40dc_aa96_8a93b76cd9a4.slice/crio-31919cf06389cdcf0472c778825f25c12c7564f2c8afb976c4ee6d9ea58b1086 WatchSource:0}: Error finding container 31919cf06389cdcf0472c778825f25c12c7564f2c8afb976c4ee6d9ea58b1086: Status 404 returned error can't find the container with id 31919cf06389cdcf0472c778825f25c12c7564f2c8afb976c4ee6d9ea58b1086 Nov 02 22:03:28.587354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:28.587318 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-21c31f" event={"ID":"b0640402-3935-40dc-aa96-8a93b76cd9a4","Type":"ContainerStarted","Data":"2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002"} Nov 02 22:03:28.587354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:28.587360 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-21c31f" event={"ID":"b0640402-3935-40dc-aa96-8a93b76cd9a4","Type":"ContainerStarted","Data":"31919cf06389cdcf0472c778825f25c12c7564f2c8afb976c4ee6d9ea58b1086"} Nov 02 22:03:28.699267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:28.699216 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-21c31f" podStartSLOduration=1.699202224 podStartE2EDuration="1.699202224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:28.698706174 +0000 UTC m=+3248.246955952" watchObservedRunningTime="2026-11-02 22:03:28.699202224 +0000 UTC m=+3248.247452005" Nov 02 22:03:29.972430 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:29.972391 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-21c31f"] Nov 02 22:03:29.977170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:29.977141 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-21c31f"] Nov 02 22:03:30.600023 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:30.599976 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-21c31f" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" containerName="test-curl-7477-21c31f" containerID="cri-o://2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002" gracePeriod=2 Nov 02 22:03:32.782617 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.782585 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-8997e5"] Nov 02 22:03:32.783281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.783263 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" containerName="test-curl-7477-21c31f" Nov 02 22:03:32.783349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.783281 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" containerName="test-curl-7477-21c31f" Nov 02 22:03:32.783411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.783402 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" containerName="test-curl-7477-21c31f" Nov 02 22:03:32.786794 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.786775 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:32.837548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.837521 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-8997e5"] Nov 02 22:03:32.850267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.850240 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:32.874628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.874572 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gw6\" (UniqueName: \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\") pod \"test-curl-7477-8997e5\" (UID: \"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae\") " pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:32.976042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.976002 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\" (UniqueName: \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\") pod \"b0640402-3935-40dc-aa96-8a93b76cd9a4\" (UID: \"b0640402-3935-40dc-aa96-8a93b76cd9a4\") " Nov 02 22:03:32.976299 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.976280 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gw6\" (UniqueName: \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\") pod \"test-curl-7477-8997e5\" (UID: \"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae\") " pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:32.978306 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:32.978282 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f" pod "b0640402-3935-40dc-aa96-8a93b76cd9a4" (UID: "b0640402-3935-40dc-aa96-8a93b76cd9a4"). InnerVolumeSpecName "kube-api-access-twd4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:33.033770 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.033739 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gw6\" (UniqueName: \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\") pod \"test-curl-7477-8997e5\" (UID: \"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae\") " pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:33.077754 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.077716 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twd4f\" (UniqueName: \"kubernetes.io/projected/b0640402-3935-40dc-aa96-8a93b76cd9a4-kube-api-access-twd4f\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:33.100723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.100700 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:33.472851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.472823 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-8997e5"] Nov 02 22:03:33.474175 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:33.474148 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe46a7c_d5c5_4f7e_8b8f_d61d029b2cae.slice/crio-ee946ebddcc1d04ce79a225a66996d0506f0f9068ce4d1f03e0caa498200256f WatchSource:0}: Error finding container ee946ebddcc1d04ce79a225a66996d0506f0f9068ce4d1f03e0caa498200256f: Status 404 returned error can't find the container with id ee946ebddcc1d04ce79a225a66996d0506f0f9068ce4d1f03e0caa498200256f Nov 02 22:03:33.622899 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.622863 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-8997e5" event={"ID":"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae","Type":"ContainerStarted","Data":"58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595"} Nov 02 22:03:33.623081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.622907 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-8997e5" event={"ID":"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae","Type":"ContainerStarted","Data":"ee946ebddcc1d04ce79a225a66996d0506f0f9068ce4d1f03e0caa498200256f"} Nov 02 22:03:33.624253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.624229 5859 generic.go:356] "Generic (PLEG): container finished" podID="b0640402-3935-40dc-aa96-8a93b76cd9a4" containerID="2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002" exitCode=137 Nov 02 22:03:33.624356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.624270 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:33.624356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.624301 5859 scope.go:122] "RemoveContainer" containerID="2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002" Nov 02 22:03:33.634509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.634488 5859 scope.go:122] "RemoveContainer" containerID="2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002" Nov 02 22:03:33.634795 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:33.634775 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002\": container with ID starting with 2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002 not found: ID does not exist" containerID="2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002" Nov 02 22:03:33.634862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.634805 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002"} err="failed to get container status \"2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002\": rpc error: code = NotFound desc = could not find container \"2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002\": container with ID starting with 2a1dbf3055c076f5ae76d9143c621f723a91531f710728af6bcc5ccd96b89002 not found: ID does not exist" Nov 02 22:03:33.737114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:33.737015 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-8997e5" podStartSLOduration=1.737002903 podStartE2EDuration="1.737002903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:33.735115634 +0000 UTC m=+3253.283365432" watchObservedRunningTime="2026-11-02 22:03:33.737002903 +0000 UTC m=+3253.285252679" Nov 02 22:03:33.786473 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:33.786433 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-21c31f\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" pod="opendatahub/test-curl-7477-21c31f" Nov 02 22:03:34.571203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:34.571173 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0640402-3935-40dc-aa96-8a93b76cd9a4" path="/var/lib/kubelet/pods/b0640402-3935-40dc-aa96-8a93b76cd9a4/volumes" Nov 02 22:03:35.078380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:35.078346 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-8997e5"] Nov 02 22:03:35.081493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:35.081465 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-8997e5"] Nov 02 22:03:35.638500 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:35.638460 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-8997e5" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" containerName="test-curl-7477-8997e5" containerID="cri-o://58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595" gracePeriod=2 Nov 02 22:03:35.690003 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:35.689942 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-8997e5\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:37.888275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:37.888243 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:38.041557 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.041472 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\" (UniqueName: \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\") pod \"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae\" (UID: \"3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae\") " Nov 02 22:03:38.043518 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.043491 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6" pod "3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" (UID: "3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae"). InnerVolumeSpecName "kube-api-access-p8gw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:38.143141 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.143113 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gw6\" (UniqueName: \"kubernetes.io/projected/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae-kube-api-access-p8gw6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:38.571760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.571728 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" path="/var/lib/kubelet/pods/3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae/volumes" Nov 02 22:03:38.658803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.658759 5859 generic.go:356] "Generic (PLEG): container finished" podID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" containerID="58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595" exitCode=137 Nov 02 22:03:38.658803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.658804 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-8997e5" Nov 02 22:03:38.659063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.658841 5859 scope.go:122] "RemoveContainer" containerID="58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595" Nov 02 22:03:38.668463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.668441 5859 scope.go:122] "RemoveContainer" containerID="58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595" Nov 02 22:03:38.668730 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:38.668703 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595\": container with ID starting with 58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595 not found: ID does not exist" containerID="58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595" Nov 02 22:03:38.668796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:38.668741 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595"} err="failed to get container status \"58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595\": rpc error: code = NotFound desc = could not find container \"58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595\": container with ID starting with 58c50d57cc14c88296e1306c84702124a1c57380c818d5c86c8f4888ee13e595 not found: ID does not exist" Nov 02 22:03:39.743608 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.743578 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:03:39.744252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.744236 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" containerName="test-curl-7477-8997e5" Nov 02 22:03:39.744308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.744253 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" containerName="test-curl-7477-8997e5" Nov 02 22:03:39.744417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.744406 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe46a7c-d5c5-4f7e-8b8f-d61d029b2cae" containerName="test-curl-7477-8997e5" Nov 02 22:03:39.749395 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.749377 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.805330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.805280 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:03:39.851904 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.851877 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8702a9-openshift-default-dockercfg-52j7c\"" Nov 02 22:03:39.863888 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.863864 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2tv\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.863994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.863908 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.863994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.863924 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864072 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864126 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864223 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864308 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864367 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.864414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.864401 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.965940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.965902 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.965989 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966043 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2tv\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966092 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966118 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966193 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966250 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966279 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966476 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966372 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.966823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966795 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.967057 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.967057 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.966971 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.967246 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.967089 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.967246 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.967126 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.969030 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.969004 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:39.969125 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:39.969107 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:40.024099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.024023 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2tv\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:40.024642 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.024611 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\") pod \"e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:40.070381 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.070347 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:40.501359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.501323 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:03:40.502088 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:40.502064 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250c73fa_bd6e_4c72_bae4_87e4c5ee11a3.slice/crio-69452f63912f6e04b0cddc737b3dfba7ab4a4c14cdcf9da4698c1789bbf1c964 WatchSource:0}: Error finding container 69452f63912f6e04b0cddc737b3dfba7ab4a4c14cdcf9da4698c1789bbf1c964: Status 404 returned error can't find the container with id 69452f63912f6e04b0cddc737b3dfba7ab4a4c14cdcf9da4698c1789bbf1c964 Nov 02 22:03:40.504546 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.504511 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:40.504644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.504578 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:40.504644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.504607 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:40.676233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.676191 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerStarted","Data":"338e7c832df6ccf603d4a4392406f7998418915fdb88d08f6b8cadda6325c09f"} Nov 02 22:03:40.676438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.676254 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerStarted","Data":"69452f63912f6e04b0cddc737b3dfba7ab4a4c14cdcf9da4698c1789bbf1c964"} Nov 02 22:03:40.794686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:40.794583 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podStartSLOduration=1.794568229 podStartE2EDuration="1.794568229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:40.792682948 +0000 UTC m=+3260.340932769" watchObservedRunningTime="2026-11-02 22:03:40.794568229 +0000 UTC m=+3260.342818086" Nov 02 22:03:41.070844 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:41.070807 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:41.076590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:41.076559 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:41.690848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:41.690804 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:41.691862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:41.691837 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:03:52.885842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.885808 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:03:52.891572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.891553 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:52.944027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.944003 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:03:52.955668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.955639 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:52.955844 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.955825 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:52.955944 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.955928 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:52.956018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.956002 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzckj\" (UniqueName: \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:52.991404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:52.991384 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-6689da-serving-cert\"" Nov 02 22:03:53.056737 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.056710 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.056863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.056779 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.056863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.056835 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzckj\" (UniqueName: \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.056986 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:53.056903 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-serving-cert: secret "maas-api-e2e-shared-a-w1-6689da-serving-cert" not found Nov 02 22:03:53.056986 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.056937 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.056986 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:53.056984 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls podName:f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03 nodeName:}" failed. No retries permitted until 2026-11-02 22:03:53.556963844 +0000 UTC m=+3273.105213616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls") pod "maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" (UID: "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03") : secret "maas-api-e2e-shared-a-w1-6689da-serving-cert" not found Nov 02 22:03:53.057531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.057511 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.059255 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.059230 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.114242 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.114218 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzckj\" (UniqueName: \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.563351 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.563315 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.565783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.565762 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:53.806641 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:53.806610 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:54.181249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.181217 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:03:54.182541 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:54.182512 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d4ffcd_b3ac_42b8_a5bf_c0c0a76f3d03.slice/crio-e5880ffe9e4d357436cc3b3132131df0e3afa8667c3f530bc14bbbd4316f0121 WatchSource:0}: Error finding container e5880ffe9e4d357436cc3b3132131df0e3afa8667c3f530bc14bbbd4316f0121: Status 404 returned error can't find the container with id e5880ffe9e4d357436cc3b3132131df0e3afa8667c3f530bc14bbbd4316f0121 Nov 02 22:03:54.788151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.788093 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" event={"ID":"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03","Type":"ContainerStarted","Data":"e5880ffe9e4d357436cc3b3132131df0e3afa8667c3f530bc14bbbd4316f0121"} Nov 02 22:03:54.902556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.902523 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:03:54.907745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.907718 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.960310 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.960279 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0f0874-openshift-default-dockercfg-jzjt8\"" Nov 02 22:03:54.980023 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.979988 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980036 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980202 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980256 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980379 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980450 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980425 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980503 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980647 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980634 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:54.980684 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:54.980663 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv8zg\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.015581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.015545 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:03:55.082220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082184 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082230 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082308 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082349 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082392 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082415 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082444 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082734 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082736 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082815 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv8zg\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.082871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.082852 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.083069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.083003 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.083193 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.083154 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.083279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.083240 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.085599 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.085567 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.085762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.085638 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.140419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.140391 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.141119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.141091 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv8zg\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\") pod \"e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.227259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.227225 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:55.660069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.660036 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:03:55.661659 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:03:55.661630 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b6b048_178e_4f77_b033_d9c38437f3a2.slice/crio-2163e15753e99cc5ee733e80c5279083396f9b64ac4100f8c20b5f6cdaec2d53 WatchSource:0}: Error finding container 2163e15753e99cc5ee733e80c5279083396f9b64ac4100f8c20b5f6cdaec2d53: Status 404 returned error can't find the container with id 2163e15753e99cc5ee733e80c5279083396f9b64ac4100f8c20b5f6cdaec2d53 Nov 02 22:03:55.664773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.664740 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:55.664852 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.664805 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:55.664852 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.664846 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:03:55.801274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.801101 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" event={"ID":"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03","Type":"ContainerStarted","Data":"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21"} Nov 02 22:03:55.801274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.801229 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:03:55.803569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.803534 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" event={"ID":"00b6b048-178e-4f77-b033-d9c38437f3a2","Type":"ContainerStarted","Data":"9ffcff078ce907c3e15f2a8245e92d9163006295af4158579e7c7c22230e8b38"} Nov 02 22:03:55.803698 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.803594 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" event={"ID":"00b6b048-178e-4f77-b033-d9c38437f3a2","Type":"ContainerStarted","Data":"2163e15753e99cc5ee733e80c5279083396f9b64ac4100f8c20b5f6cdaec2d53"} Nov 02 22:03:55.915729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:55.915629 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" podStartSLOduration=3.002038405 podStartE2EDuration="3.915612955s" totalImagesPullingTime="913.57455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:55.914887402 +0000 UTC m=+3275.463137225" watchObservedRunningTime="2026-11-02 22:03:55.915612955 +0000 UTC m=+3275.463862736" Nov 02 22:03:55.952990 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:03:55.951099 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:03:56.032651 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:56.032594 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podStartSLOduration=2.032577446 podStartE2EDuration="2.032577446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:56.030269295 +0000 UTC m=+3275.578519076" watchObservedRunningTime="2026-11-02 22:03:56.032577446 +0000 UTC m=+3275.580827227" Nov 02 22:03:56.228452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:56.228368 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:03:56.229915 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:56.229886 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:03:56.230091 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:56.229997 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:03:57.230990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:57.230272 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:03:57.230990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:57.230327 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:03:58.231009 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:58.230941 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:03:58.231628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:58.231024 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:03:59.228976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:59.228358 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:03:59.228976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:03:59.228423 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:04:00.230448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:00.230353 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:04:00.230448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:00.230407 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:04:01.230411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:01.230371 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 22:04:01.230652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:01.230430 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 22:04:01.842234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:01.842205 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:04:02.232280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:02.232191 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:04:02.232740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:02.232659 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:04:02.233279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:02.233261 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:04:21.011369 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.011337 5859 generic.go:356] "Generic (PLEG): container finished" podID="b724c917-6f74-45a9-b517-bf5d039c678a" containerID="f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923" exitCode=0 Nov 02 22:04:21.011752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.011411 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerDied","Data":"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923"} Nov 02 22:04:21.011933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.011918 5859 scope.go:122] "RemoveContainer" containerID="f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923" Nov 02 22:04:21.013721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.013688 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:21.013846 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.013769 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:21.013846 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:21.013818 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:04:22.038965 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:22.038922 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerStarted","Data":"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9"} Nov 02 22:04:22.392593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:22.392542 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:04:22.393792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:22.393770 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:22.393879 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:22.393816 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:23.392024 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:23.391942 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:23.392421 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:23.392040 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:24.392416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:24.392377 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:24.392834 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:24.392445 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:25.272326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:25.272287 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:04:25.392207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:25.392166 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:25.392402 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:25.392242 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:25.955848 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:04:25.955813 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:04:26.392176 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:26.392144 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:26.392409 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:26.392194 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:27.392337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:27.392301 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 22:04:27.392756 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:27.392357 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 22:04:28.396743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:28.396711 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:04:28.397619 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:04:28.397599 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:04:55.956174 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:04:55.956135 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:05:25.945893 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:05:25.945810 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:05:36.309277 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:05:36.308610 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@18-45150-0.service\": RecentStats: unable to find data in memory cache]" Nov 02 22:05:36.581886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.581854 5859 generic.go:356] "Generic (PLEG): container finished" podID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerID="338e7c832df6ccf603d4a4392406f7998418915fdb88d08f6b8cadda6325c09f" exitCode=0 Nov 02 22:05:36.582074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.581925 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerDied","Data":"338e7c832df6ccf603d4a4392406f7998418915fdb88d08f6b8cadda6325c09f"} Nov 02 22:05:36.582336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.582321 5859 scope.go:122] "RemoveContainer" containerID="338e7c832df6ccf603d4a4392406f7998418915fdb88d08f6b8cadda6325c09f" Nov 02 22:05:36.584047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.584015 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:36.584172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.584101 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:36.584172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:36.584159 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:37.590582 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:37.590552 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerStarted","Data":"c46780c0768213975b07af91c6f1998098ccea9c3c4a2f77125658175f1be7bb"} Nov 02 22:05:38.070893 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:38.070866 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:05:38.072302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:38.072274 5859 patch_prober.go:29] interesting pod/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 22:05:38.072365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:38.072323 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 22:05:38.656535 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:38.656504 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:05:39.071651 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:39.071614 5859 patch_prober.go:29] interesting pod/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 22:05:39.071852 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:39.071667 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 22:05:40.071054 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:40.071014 5859 patch_prober.go:29] interesting pod/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 22:05:40.071463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:40.071071 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 22:05:41.070919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:41.070883 5859 patch_prober.go:29] interesting pod/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 22:05:41.071103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:41.070939 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 22:05:41.691264 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:41.691236 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:05:42.071340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:42.071300 5859 patch_prober.go:29] interesting pod/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 22:05:42.071820 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:42.071353 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 22:05:43.075742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:43.075712 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:05:43.076993 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:43.076943 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:05:53.688571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:53.688505 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" podUID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" containerName="main" containerID="cri-o://5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2" gracePeriod=45 Nov 02 22:05:53.965535 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:53.965506 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:05:54.061639 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.061598 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.061639 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.061644 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.061942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.061681 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.061942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.061876 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\" (UniqueName: \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.061942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.061917 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:54.062157 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:54.062243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062224 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.062322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062309 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\") pod \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\" (UID: \"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3\") " Nov 02 22:05:54.062649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062593 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:54.062880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062849 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.062880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062879 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.063064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.062891 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.064407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.064384 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:54.064591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.064564 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:54.064652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.064610 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g" pod "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" (UID: "4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3"). InnerVolumeSpecName "kube-api-access-cb84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:54.163902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.163866 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.163902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.163899 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.163902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.163910 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb84g\" (UniqueName: \"kubernetes.io/projected/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3-kube-api-access-cb84g\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:54.742203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.742167 5859 generic.go:356] "Generic (PLEG): container finished" podID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" containerID="5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2" exitCode=0 Nov 02 22:05:54.742666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.742210 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" Nov 02 22:05:54.742666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.742263 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" event={"ID":"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3","Type":"ContainerDied","Data":"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2"} Nov 02 22:05:54.742666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.742300 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq" event={"ID":"4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3","Type":"ContainerDied","Data":"79f6405c2cdb2c3a4e52e10fa6c3d807c6a5e8489c72291e601c1ddc4d6a5702"} Nov 02 22:05:54.742666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.742324 5859 scope.go:122] "RemoveContainer" containerID="5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2" Nov 02 22:05:54.752718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.752697 5859 scope.go:122] "RemoveContainer" containerID="5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2" Nov 02 22:05:54.753083 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:05:54.753059 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2\": container with ID starting with 5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2 not found: ID does not exist" containerID="5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2" Nov 02 22:05:54.753208 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.753091 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2"} err="failed to get container status \"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2\": rpc error: code = NotFound desc = could not find container \"5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2\": container with ID starting with 5623dd4fec5fac59456e6ed2370bd59f2d90f38dd27df3ccbf828a64110f81e2 not found: ID does not exist" Nov 02 22:05:54.907920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.907830 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:05:54.957975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:54.957916 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/auth-test-model-58ca7f-kserve-bbf485c5c-gpdfq"] Nov 02 22:05:55.946259 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:05:55.946216 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:05:56.571414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:05:56.571375 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" path="/var/lib/kubelet/pods/4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3/volumes" Nov 02 22:06:25.945792 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:06:25.945753 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:06:36.927260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.927176 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:06:36.927775 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.927761 5859 cpu_manager.go:414] "containerMap: removing container" podUID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" containerName="main" Nov 02 22:06:36.927827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.927776 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" containerName="main" Nov 02 22:06:36.927906 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.927897 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4711eddf-5a4e-4fe1-91d1-9b88ce6bbde3" containerName="main" Nov 02 22:06:36.931498 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.931480 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:36.985275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:36.985250 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:06:37.014173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.014148 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.014326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.014205 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.014326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.014314 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggf6q\" (UniqueName: \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.014473 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.014425 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.032310 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.032284 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9981d2-serving-cert\"" Nov 02 22:06:37.115328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.115295 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.115561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.115387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.115561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.115419 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggf6q\" (UniqueName: \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.115561 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:06:37.115520 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-serving-cert: secret "maas-api-e2e-extmod-a-9981d2-serving-cert" not found Nov 02 22:06:37.115561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.115539 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.115851 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:06:37.115587 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls podName:70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4 nodeName:}" failed. No retries permitted until 2026-11-02 22:06:37.615568951 +0000 UTC m=+3437.163818726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls") pod "maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" (UID: "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4") : secret "maas-api-e2e-extmod-a-9981d2-serving-cert" not found Nov 02 22:06:37.116283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.116250 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.117900 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.117868 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.173142 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.173115 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggf6q\" (UniqueName: \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.622779 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.622739 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.625223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.625202 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:37.848447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:37.848417 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:38.226047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:38.226020 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:06:38.226876 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:06:38.226844 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ee86d3_413e_4ad9_aaf0_bd3dc57fc9b4.slice/crio-26c3e23a7489c6c3ae08e7e4e87ae18f3ab255ce7f36be49c8dc28af443d064a WatchSource:0}: Error finding container 26c3e23a7489c6c3ae08e7e4e87ae18f3ab255ce7f36be49c8dc28af443d064a: Status 404 returned error can't find the container with id 26c3e23a7489c6c3ae08e7e4e87ae18f3ab255ce7f36be49c8dc28af443d064a Nov 02 22:06:38.228330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:38.228307 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:06:39.037263 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:39.037230 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" event={"ID":"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4","Type":"ContainerStarted","Data":"26c3e23a7489c6c3ae08e7e4e87ae18f3ab255ce7f36be49c8dc28af443d064a"} Nov 02 22:06:40.045747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:40.045700 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" event={"ID":"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4","Type":"ContainerStarted","Data":"45157dad145e8d666d7de5b6aef028e0b3e6cd4804ad3be51a3fdd28c62a6953"} Nov 02 22:06:40.046168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:40.045770 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:40.165686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:40.165640 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" podStartSLOduration=3.264102161 podStartE2EDuration="4.165625271s" totalImagesPullingTime="901.52311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:40.16256959 +0000 UTC m=+3439.710819372" watchObservedRunningTime="2026-11-02 22:06:40.165625271 +0000 UTC m=+3439.713875052" Nov 02 22:06:46.063266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:46.063235 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:06:52.552376 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.552331 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:06:52.556659 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.556643 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.611537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.611499 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:06:52.658298 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.658265 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-5ddbd0-serving-cert\"" Nov 02 22:06:52.709141 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.709116 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.709317 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.709291 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.709451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.709405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.709451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.709447 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jh6z\" (UniqueName: \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.810898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.810829 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.810898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.810866 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jh6z\" (UniqueName: \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.811079 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.810903 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.811079 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.811001 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.811536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.811516 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.813407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.813385 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.813512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.813455 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.867699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.867676 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jh6z\" (UniqueName: \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\") pod \"maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:52.870525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:52.870497 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:53.243787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:53.243755 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:06:53.244797 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:06:53.244771 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd023b45f_ebfd_45b0_b8f5_0b20aff0cf55.slice/crio-01b008d3f43190a8f0161d584c5d9c0fa2919cc59ee6565dc2d70a1beab01482 WatchSource:0}: Error finding container 01b008d3f43190a8f0161d584c5d9c0fa2919cc59ee6565dc2d70a1beab01482: Status 404 returned error can't find the container with id 01b008d3f43190a8f0161d584c5d9c0fa2919cc59ee6565dc2d70a1beab01482 Nov 02 22:06:54.153310 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:54.153277 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" event={"ID":"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55","Type":"ContainerStarted","Data":"01b008d3f43190a8f0161d584c5d9c0fa2919cc59ee6565dc2d70a1beab01482"} Nov 02 22:06:55.162266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:55.162232 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" event={"ID":"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55","Type":"ContainerStarted","Data":"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387"} Nov 02 22:06:55.162643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:55.162321 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:06:55.278969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:06:55.278905 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" podStartSLOduration=2.404879449 podStartE2EDuration="3.278893291s" totalImagesPullingTime="874.013842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:55.276234065 +0000 UTC m=+3454.824483867" watchObservedRunningTime="2026-11-02 22:06:55.278893291 +0000 UTC m=+3454.827143091" Nov 02 22:06:55.945333 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:06:55.945299 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:07:01.174662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:07:01.174635 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:07:25.946803 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:07:25.946748 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:07:55.947851 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:07:55.947805 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:08:03.320999 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:03.320887 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:08:18.342287 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.342231 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" podUID="c3719044-505f-418b-8dbd-c9686a9e2c0a" containerName="main" containerID="cri-o://73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82" gracePeriod=45 Nov 02 22:08:18.619188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.619122 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:08:18.731262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731225 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.731455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731283 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.731532 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731457 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.731602 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731540 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.731602 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731564 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.731602 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.731587 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\" (UniqueName: \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\") pod \"c3719044-505f-418b-8dbd-c9686a9e2c0a\" (UID: \"c3719044-505f-418b-8dbd-c9686a9e2c0a\") " Nov 02 22:08:18.732662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.732607 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:18.732662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.732620 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:18.732885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.732827 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:18.737989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.734937 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:18.737989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.734939 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "kube-api-access-hkwsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:18.737989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.735142 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs" pod "c3719044-505f-418b-8dbd-c9686a9e2c0a" (UID: "c3719044-505f-418b-8dbd-c9686a9e2c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:18.755536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.755506 5859 generic.go:356] "Generic (PLEG): container finished" podID="c3719044-505f-418b-8dbd-c9686a9e2c0a" containerID="73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82" exitCode=0 Nov 02 22:08:18.755660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.755553 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" Nov 02 22:08:18.755660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.755591 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" event={"ID":"c3719044-505f-418b-8dbd-c9686a9e2c0a","Type":"ContainerDied","Data":"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82"} Nov 02 22:08:18.755660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.755633 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt" event={"ID":"c3719044-505f-418b-8dbd-c9686a9e2c0a","Type":"ContainerDied","Data":"2471aa67b08589bdabf59a01112cb2378ae76695f2d3e412a2c559a6653ef865"} Nov 02 22:08:18.755660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.755651 5859 scope.go:122] "RemoveContainer" containerID="73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82" Nov 02 22:08:18.772980 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.771936 5859 scope.go:122] "RemoveContainer" containerID="73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82" Nov 02 22:08:18.773312 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:18.773288 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82\": container with ID starting with 73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82 not found: ID does not exist" containerID="73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82" Nov 02 22:08:18.773410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.773316 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82"} err="failed to get container status \"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82\": rpc error: code = NotFound desc = could not find container \"73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82\": container with ID starting with 73feabf3c0906c82628c90f2f05d272ca33837f04a3510783f642dde5d4bff82 not found: ID does not exist" Nov 02 22:08:18.833154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833118 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3719044-505f-418b-8dbd-c9686a9e2c0a-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.833154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833152 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.833384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833165 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.833384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833183 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwsc\" (UniqueName: \"kubernetes.io/projected/c3719044-505f-418b-8dbd-c9686a9e2c0a-kube-api-access-hkwsc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.833384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833196 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.833384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.833207 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3719044-505f-418b-8dbd-c9686a9e2c0a-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:18.920860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.920777 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:08:18.923621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:18.923585 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-06e461/auth-test-model-06e461-kserve-8468c86b45-vn4qt"] Nov 02 22:08:20.572595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:20.572559 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3719044-505f-418b-8dbd-c9686a9e2c0a" path="/var/lib/kubelet/pods/c3719044-505f-418b-8dbd-c9686a9e2c0a/volumes" Nov 02 22:08:25.946634 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:25.946599 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:08:30.279886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:30.279845 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:08:30.385728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:30.385697 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:08:35.281069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.281022 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" podUID="aeac2e05-049b-457a-ad96-ab5b83e33d62" containerName="payload-processing" containerID="cri-o://7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c" gracePeriod=25 Nov 02 22:08:35.386834 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.386787 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" podUID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" containerName="payload-pre-processing" containerID="cri-o://1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748" gracePeriod=25 Nov 02 22:08:35.573177 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.573151 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:08:35.655708 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.655685 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:08:35.663821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.663802 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\" (UniqueName: \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\") pod \"aeac2e05-049b-457a-ad96-ab5b83e33d62\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " Nov 02 22:08:35.664026 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.664008 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\") pod \"aeac2e05-049b-457a-ad96-ab5b83e33d62\" (UID: \"aeac2e05-049b-457a-ad96-ab5b83e33d62\") " Nov 02 22:08:35.664412 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.664389 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume" pod "aeac2e05-049b-457a-ad96-ab5b83e33d62" (UID: "aeac2e05-049b-457a-ad96-ab5b83e33d62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:35.664518 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.664504 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeac2e05-049b-457a-ad96-ab5b83e33d62-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:35.665808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.665786 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh" pod "aeac2e05-049b-457a-ad96-ab5b83e33d62" (UID: "aeac2e05-049b-457a-ad96-ab5b83e33d62"). InnerVolumeSpecName "kube-api-access-fwkqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:35.765459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.765429 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\" (UniqueName: \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\") pod \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " Nov 02 22:08:35.765650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.765478 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\") pod \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\" (UID: \"1c160b12-f67c-46fc-83a9-8ff5ce482ff9\") " Nov 02 22:08:35.765945 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.765912 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume" pod "1c160b12-f67c-46fc-83a9-8ff5ce482ff9" (UID: "1c160b12-f67c-46fc-83a9-8ff5ce482ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:35.766163 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.766144 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkqh\" (UniqueName: \"kubernetes.io/projected/aeac2e05-049b-457a-ad96-ab5b83e33d62-kube-api-access-fwkqh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:35.766226 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.766169 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:35.767729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.767704 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6" pod "1c160b12-f67c-46fc-83a9-8ff5ce482ff9" (UID: "1c160b12-f67c-46fc-83a9-8ff5ce482ff9"). InnerVolumeSpecName "kube-api-access-n6zl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:35.867046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.867012 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6zl6\" (UniqueName: \"kubernetes.io/projected/1c160b12-f67c-46fc-83a9-8ff5ce482ff9-kube-api-access-n6zl6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:35.885330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.885297 5859 generic.go:356] "Generic (PLEG): container finished" podID="aeac2e05-049b-457a-ad96-ab5b83e33d62" containerID="7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c" exitCode=0 Nov 02 22:08:35.885481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.885349 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" Nov 02 22:08:35.885481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.885390 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" event={"ID":"aeac2e05-049b-457a-ad96-ab5b83e33d62","Type":"ContainerDied","Data":"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c"} Nov 02 22:08:35.885481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.885464 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb" event={"ID":"aeac2e05-049b-457a-ad96-ab5b83e33d62","Type":"ContainerDied","Data":"a1184927b569b70809a0fc5c06922a3eade25c531bc77d9cf0fe6b7a0948f5d4"} Nov 02 22:08:35.885609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.885499 5859 scope.go:122] "RemoveContainer" containerID="7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c" Nov 02 22:08:35.887207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.887185 5859 generic.go:356] "Generic (PLEG): container finished" podID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" containerID="1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748" exitCode=0 Nov 02 22:08:35.887339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.887239 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" Nov 02 22:08:35.887339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.887256 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" event={"ID":"1c160b12-f67c-46fc-83a9-8ff5ce482ff9","Type":"ContainerDied","Data":"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748"} Nov 02 22:08:35.887463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.887385 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4" event={"ID":"1c160b12-f67c-46fc-83a9-8ff5ce482ff9","Type":"ContainerDied","Data":"df65e80fd1b3533abad300ed3cc94e67d8925aeb0e9a7cb80565f87293d2f136"} Nov 02 22:08:35.897878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.897858 5859 scope.go:122] "RemoveContainer" containerID="7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c" Nov 02 22:08:35.898150 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:35.898122 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c\": container with ID starting with 7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c not found: ID does not exist" containerID="7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c" Nov 02 22:08:35.898251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.898159 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c"} err="failed to get container status \"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c\": rpc error: code = NotFound desc = could not find container \"7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c\": container with ID starting with 7991297d6561af7e65c5fc6b0200a26e0daf4e1a2f232486268a0d7d74c5dd0c not found: ID does not exist" Nov 02 22:08:35.898251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.898183 5859 scope.go:122] "RemoveContainer" containerID="1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748" Nov 02 22:08:35.907389 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.907372 5859 scope.go:122] "RemoveContainer" containerID="1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748" Nov 02 22:08:35.907625 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:35.907606 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748\": container with ID starting with 1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748 not found: ID does not exist" containerID="1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748" Nov 02 22:08:35.907673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:35.907632 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748"} err="failed to get container status \"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748\": rpc error: code = NotFound desc = could not find container \"1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748\": container with ID starting with 1ff7dc7397a8e2f5cf2627ceb4d67758a69644ca734cf9f92d7b716afbe4c748 not found: ID does not exist" Nov 02 22:08:36.053149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.053113 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:08:36.056633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.056606 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6689da-86b48d4cd7-92hz4"] Nov 02 22:08:36.165898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.165804 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:08:36.168639 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.168612 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6689da-55c9b99ff-gw6rb"] Nov 02 22:08:36.571702 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.571669 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" path="/var/lib/kubelet/pods/1c160b12-f67c-46fc-83a9-8ff5ce482ff9/volumes" Nov 02 22:08:36.572317 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:36.572296 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeac2e05-049b-457a-ad96-ab5b83e33d62" path="/var/lib/kubelet/pods/aeac2e05-049b-457a-ad96-ab5b83e33d62/volumes" Nov 02 22:08:39.487442 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:39.487410 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:08:46.924395 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.924360 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:08:46.924988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.924974 5859 cpu_manager.go:414] "containerMap: removing container" podUID="aeac2e05-049b-457a-ad96-ab5b83e33d62" containerName="payload-processing" Nov 02 22:08:46.925106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.924989 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeac2e05-049b-457a-ad96-ab5b83e33d62" containerName="payload-processing" Nov 02 22:08:46.925106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925027 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c3719044-505f-418b-8dbd-c9686a9e2c0a" containerName="main" Nov 02 22:08:46.925106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925032 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3719044-505f-418b-8dbd-c9686a9e2c0a" containerName="main" Nov 02 22:08:46.925106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925043 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" containerName="payload-pre-processing" Nov 02 22:08:46.925106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925048 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" containerName="payload-pre-processing" Nov 02 22:08:46.925267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925145 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3719044-505f-418b-8dbd-c9686a9e2c0a" containerName="main" Nov 02 22:08:46.925267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925155 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c160b12-f67c-46fc-83a9-8ff5ce482ff9" containerName="payload-pre-processing" Nov 02 22:08:46.925267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.925165 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeac2e05-049b-457a-ad96-ab5b83e33d62" containerName="payload-processing" Nov 02 22:08:46.930342 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.930322 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:46.981443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:46.981407 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:08:47.005922 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.005877 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thvd\" (UniqueName: \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.006188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.006071 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.049241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.049205 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-363e15-dockercfg-6ndrh\"" Nov 02 22:08:47.049626 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.049604 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-363e15\"" Nov 02 22:08:47.107103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.107072 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thvd\" (UniqueName: \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.107281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.107190 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.107884 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.107859 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.164642 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.164611 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thvd\" (UniqueName: \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\") pod \"payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.249595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.249492 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:47.637980 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.637937 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:08:47.639817 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:08:47.639790 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c5765c_609b_4dfd_9438_869af50a639b.slice/crio-399a7bf4a67930a64aa946d88412359708e9c85a0694c9bf87938937a302d2e1 WatchSource:0}: Error finding container 399a7bf4a67930a64aa946d88412359708e9c85a0694c9bf87938937a302d2e1: Status 404 returned error can't find the container with id 399a7bf4a67930a64aa946d88412359708e9c85a0694c9bf87938937a302d2e1 Nov 02 22:08:47.973010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.972911 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" event={"ID":"b0c5765c-609b-4dfd-9438-869af50a639b","Type":"ContainerStarted","Data":"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b"} Nov 02 22:08:47.973010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.972986 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" event={"ID":"b0c5765c-609b-4dfd-9438-869af50a639b","Type":"ContainerStarted","Data":"399a7bf4a67930a64aa946d88412359708e9c85a0694c9bf87938937a302d2e1"} Nov 02 22:08:47.973438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:47.973014 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:08:48.091511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:48.091449 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" podStartSLOduration=2.091432263 podStartE2EDuration="2.091432263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:48.087807724 +0000 UTC m=+3567.636057523" watchObservedRunningTime="2026-11-02 22:08:48.091432263 +0000 UTC m=+3567.639682045" Nov 02 22:08:54.508208 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.508131 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerName="main" containerID="cri-o://1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1" gracePeriod=45 Nov 02 22:08:54.573703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.573664 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": dial tcp 10.134.0.158:8000: connect: connection refused" Nov 02 22:08:54.793903 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.793861 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:08:54.912069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912032 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912198 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912254 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912409 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912344 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:54.912460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912404 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912448 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912502 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\" (UniqueName: \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\") pod \"74b1ce7c-f811-45ac-80be-38958d3386d3\" (UID: \"74b1ce7c-f811-45ac-80be-38958d3386d3\") " Nov 02 22:08:54.912660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912615 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:54.912778 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.912756 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:54.913074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.913047 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.913074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.913073 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.913487 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.913086 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:54.915482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.915455 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:54.918127 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.918098 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "kube-api-access-s47vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:54.921751 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:54.921726 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs" pod "74b1ce7c-f811-45ac-80be-38958d3386d3" (UID: "74b1ce7c-f811-45ac-80be-38958d3386d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:55.014455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.014365 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47vg\" (UniqueName: \"kubernetes.io/projected/74b1ce7c-f811-45ac-80be-38958d3386d3-kube-api-access-s47vg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:55.014455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.014399 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b1ce7c-f811-45ac-80be-38958d3386d3-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:55.014455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.014412 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b1ce7c-f811-45ac-80be-38958d3386d3-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:55.023513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.023483 5859 generic.go:356] "Generic (PLEG): container finished" podID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerID="1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1" exitCode=0 Nov 02 22:08:55.023657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.023529 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" Nov 02 22:08:55.023657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.023567 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" event={"ID":"74b1ce7c-f811-45ac-80be-38958d3386d3","Type":"ContainerDied","Data":"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1"} Nov 02 22:08:55.023657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.023611 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d" event={"ID":"74b1ce7c-f811-45ac-80be-38958d3386d3","Type":"ContainerDied","Data":"c5fafb63a3888bfca4851366c7c89feb13c31d272965993f3d674ac382f5fdc9"} Nov 02 22:08:55.023657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.023628 5859 scope.go:122] "RemoveContainer" containerID="1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1" Nov 02 22:08:55.035404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.035386 5859 scope.go:122] "RemoveContainer" containerID="1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1" Nov 02 22:08:55.035673 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:55.035652 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1\": container with ID starting with 1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1 not found: ID does not exist" containerID="1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1" Nov 02 22:08:55.035721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.035682 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1"} err="failed to get container status \"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1\": rpc error: code = NotFound desc = could not find container \"1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1\": container with ID starting with 1eeeb1de20b94ebf8bf18df2bf75c8962d575296fbc18ae2f4447e9f53f702d1 not found: ID does not exist" Nov 02 22:08:55.188477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.188448 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:08:55.191113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:55.191086 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c7e71f/auth-test-model-c7e71f-kserve-757bd54d9c-v6f7d"] Nov 02 22:08:55.946815 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:08:55.946775 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:08:56.571970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:56.571915 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" path="/var/lib/kubelet/pods/74b1ce7c-f811-45ac-80be-38958d3386d3/volumes" Nov 02 22:08:58.983766 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:08:58.983731 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:09:25.945472 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:09:25.945433 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:09:46.163202 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.163166 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:09:46.163796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.163782 5859 cpu_manager.go:414] "containerMap: removing container" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerName="main" Nov 02 22:09:46.163847 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.163797 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerName="main" Nov 02 22:09:46.163937 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.163928 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b1ce7c-f811-45ac-80be-38958d3386d3" containerName="main" Nov 02 22:09:46.167730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.167714 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.213720 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.213686 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:09:46.217975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.217932 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.231783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.231762 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\"" Nov 02 22:09:46.254808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.254757 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.254983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.254857 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.254983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.254897 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255133 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255056 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr77\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255133 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255099 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255193 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255220 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255248 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255369 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255425 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255396 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255425 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255418 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255496 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255459 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxckq\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255565 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255542 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255661 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255649 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255680 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.255733 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255706 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255775 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.255824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.255804 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.266462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.266435 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:09:46.318457 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.318417 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:09:46.357408 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357369 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr77\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357408 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357413 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357543 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.357677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357581 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357615 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.357677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357664 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357683 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357715 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357751 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxckq\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357788 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357796 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.357996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.357985 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358092 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358118 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358155 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358197 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358242 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.358328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358273 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358338 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358381 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358407 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.358663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358466 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358496 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358986 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358824 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.358986 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.358865 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.359243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.359214 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.359370 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.359218 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.359370 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.359274 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.360346 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.360325 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.360448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.360431 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.360605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.360580 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.361124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.361103 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.415048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.414972 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr77\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.415218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.415169 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxckq\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.415299 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.415278 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-z8bpp\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.415472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.415450 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rpfpj\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.482843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.482804 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:46.544080 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.543197 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:46.948163 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.948136 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:09:46.950134 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:09:46.950107 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5a4e23_2b6f_48e6_970b_1eacc5b5cf23.slice/crio-e34c8a6bad2e391601699f17e66c6429d4b451f92c6ee6f697c3156c0cef5a6c WatchSource:0}: Error finding container e34c8a6bad2e391601699f17e66c6429d4b451f92c6ee6f697c3156c0cef5a6c: Status 404 returned error can't find the container with id e34c8a6bad2e391601699f17e66c6429d4b451f92c6ee6f697c3156c0cef5a6c Nov 02 22:09:46.952899 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.952865 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:46.952995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.952930 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:46.952995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:46.952982 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:47.003103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.002908 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:09:47.006500 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:09:47.006469 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3e8f0c_0f0e_49aa_b36e_0789cc3c7222.slice/crio-4d4c655feabf9e7aa929b74410f1798568eaaef162206f8ec9c8ab7e2e4ff751 WatchSource:0}: Error finding container 4d4c655feabf9e7aa929b74410f1798568eaaef162206f8ec9c8ab7e2e4ff751: Status 404 returned error can't find the container with id 4d4c655feabf9e7aa929b74410f1798568eaaef162206f8ec9c8ab7e2e4ff751 Nov 02 22:09:47.009443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.009409 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:47.009549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.009495 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:47.009549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.009541 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:47.382973 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.382638 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" event={"ID":"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222","Type":"ContainerStarted","Data":"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88"} Nov 02 22:09:47.382973 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.382704 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" event={"ID":"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222","Type":"ContainerStarted","Data":"4d4c655feabf9e7aa929b74410f1798568eaaef162206f8ec9c8ab7e2e4ff751"} Nov 02 22:09:47.386748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.386695 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" event={"ID":"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23","Type":"ContainerStarted","Data":"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c"} Nov 02 22:09:47.386945 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.386775 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" event={"ID":"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23","Type":"ContainerStarted","Data":"e34c8a6bad2e391601699f17e66c6429d4b451f92c6ee6f697c3156c0cef5a6c"} Nov 02 22:09:47.482990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.482941 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:47.504094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.504029 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podStartSLOduration=1.5039351399999998 podStartE2EDuration="1.50393514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:47.500840818 +0000 UTC m=+3627.049090603" watchObservedRunningTime="2026-11-02 22:09:47.50393514 +0000 UTC m=+3627.052184923" Nov 02 22:09:47.544426 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.544391 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:47.623283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:47.619430 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podStartSLOduration=1.619411399 podStartE2EDuration="1.619411399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:47.617931579 +0000 UTC m=+3627.166181388" watchObservedRunningTime="2026-11-02 22:09:47.619411399 +0000 UTC m=+3627.167661182" Nov 02 22:09:48.484547 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:48.484512 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:48.484973 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:48.484563 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:48.555066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:48.554901 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:48.555066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:48.554996 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:49.483477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:49.483434 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:49.483678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:49.483503 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:49.552502 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:49.552458 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:49.553137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:49.553102 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:50.484050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:50.484013 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:50.484266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:50.484070 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:50.547976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:50.546988 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:50.547976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:50.547061 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:51.483719 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:51.483679 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:51.484357 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:51.484325 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:51.547977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:51.547505 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:51.547977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:51.547622 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:52.483703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:52.483670 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:52.483910 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:52.483718 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:52.548975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:52.546591 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:52.548975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:52.546671 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:53.484138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:53.484095 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:53.484600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:53.484151 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:53.544643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:53.544605 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:53.544803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:53.544680 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:54.483461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:54.483419 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:09:54.483635 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:54.483499 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:09:54.544682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:54.544643 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 22:09:54.545119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:54.544709 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 22:09:55.707159 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.707120 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:55.707743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.707718 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:55.708998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.708974 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:09:55.734636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.734588 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:55.735566 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.735493 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:55.736413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:55.736368 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:09:55.945205 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:09:55.945163 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:09:57.405394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:57.405364 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:09:57.405830 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:09:57.405569 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" containerID="cri-o://9ffcff078ce907c3e15f2a8245e92d9163006295af4158579e7c7c22230e8b38" gracePeriod=30 Nov 02 22:10:02.234753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.234706 5859 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:10:02.235189 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.234778 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:10:02.557165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.557132 5859 generic.go:356] "Generic (PLEG): container finished" podID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerID="9ffcff078ce907c3e15f2a8245e92d9163006295af4158579e7c7c22230e8b38" exitCode=0 Nov 02 22:10:02.557344 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.557200 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" event={"ID":"00b6b048-178e-4f77-b033-d9c38437f3a2","Type":"ContainerDied","Data":"9ffcff078ce907c3e15f2a8245e92d9163006295af4158579e7c7c22230e8b38"} Nov 02 22:10:02.691913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.691884 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:10:02.785203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785128 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785171 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785280 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785362 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785397 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785420 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785440 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:02.785760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785483 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785522 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785525 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:02.785760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785574 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\") pod \"00b6b048-178e-4f77-b033-d9c38437f3a2\" (UID: \"00b6b048-178e-4f77-b033-d9c38437f3a2\") " Nov 02 22:10:02.785975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785771 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:02.785975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.785820 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:02.786076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786008 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:02.786252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786232 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.786323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786256 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.786323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786271 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.786323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786285 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.786323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.786300 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b6b048-178e-4f77-b033-d9c38437f3a2-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.787763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.787731 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:02.788002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.787931 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "kube-api-access-zv8zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:02.788158 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.788138 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:10:02.788265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.788249 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token" pod "00b6b048-178e-4f77-b033-d9c38437f3a2" (UID: "00b6b048-178e-4f77-b033-d9c38437f3a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:02.888286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.888247 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.888286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.888287 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.888485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.888302 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b6b048-178e-4f77-b033-d9c38437f3a2-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:02.888485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:02.888317 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv8zg\" (UniqueName: \"kubernetes.io/projected/00b6b048-178e-4f77-b033-d9c38437f3a2-kube-api-access-zv8zg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:03.565771 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:03.565740 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" Nov 02 22:10:03.566245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:03.565739 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx" event={"ID":"00b6b048-178e-4f77-b033-d9c38437f3a2","Type":"ContainerDied","Data":"2163e15753e99cc5ee733e80c5279083396f9b64ac4100f8c20b5f6cdaec2d53"} Nov 02 22:10:03.566245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:03.565881 5859 scope.go:122] "RemoveContainer" containerID="9ffcff078ce907c3e15f2a8245e92d9163006295af4158579e7c7c22230e8b38" Nov 02 22:10:03.784296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:03.784259 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:10:03.834441 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:03.834401 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0f0874-openshift-default-84fb9cfcf9-g7ntx"] Nov 02 22:10:04.572241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:04.572204 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" path="/var/lib/kubelet/pods/00b6b048-178e-4f77-b033-d9c38437f3a2/volumes" Nov 02 22:10:08.185376 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.185329 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:10:08.186800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.186770 5859 cpu_manager.go:414] "containerMap: removing container" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" Nov 02 22:10:08.186800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.186798 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" Nov 02 22:10:08.187230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.187211 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b6b048-178e-4f77-b033-d9c38437f3a2" containerName="istio-proxy" Nov 02 22:10:08.194212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.194179 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.243008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.242328 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:10:08.247466 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.247435 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-363e15-serving-cert\"" Nov 02 22:10:08.252224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.252185 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.252402 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.252304 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.252485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.252453 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.252554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.252521 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdhl\" (UniqueName: \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.353801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.353750 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.354023 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.353847 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.354023 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.354004 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.354178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.354098 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdhl\" (UniqueName: \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.354178 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:10:08.354007 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-363e15-serving-cert: secret "maas-api-e2e-autores-363e15-serving-cert" not found Nov 02 22:10:08.354297 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:10:08.354214 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls podName:c8ab9390-8991-4fba-bd7e-d476fe1af0b4 nodeName:}" failed. No retries permitted until 2026-11-02 22:10:08.854189222 +0000 UTC m=+3648.402438982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls") pod "maas-api-e2e-autores-363e15-79f7487d79-wfqht" (UID: "c8ab9390-8991-4fba-bd7e-d476fe1af0b4") : secret "maas-api-e2e-autores-363e15-serving-cert" not found Nov 02 22:10:08.354536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.354512 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.356613 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.356583 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.411545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.411509 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdhl\" (UniqueName: \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.860352 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.860310 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:08.862832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:08.862799 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") pod \"maas-api-e2e-autores-363e15-79f7487d79-wfqht\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:09.110685 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:09.110581 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:09.488088 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:09.488062 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:10:09.490402 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:10:09.490364 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ab9390_8991_4fba_bd7e_d476fe1af0b4.slice/crio-e9bd142ea2d704a10a3e43ee9682237798683df0512227d4120b21a4f5303cd5 WatchSource:0}: Error finding container e9bd142ea2d704a10a3e43ee9682237798683df0512227d4120b21a4f5303cd5: Status 404 returned error can't find the container with id e9bd142ea2d704a10a3e43ee9682237798683df0512227d4120b21a4f5303cd5 Nov 02 22:10:09.617711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:09.617670 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" event={"ID":"c8ab9390-8991-4fba-bd7e-d476fe1af0b4","Type":"ContainerStarted","Data":"e9bd142ea2d704a10a3e43ee9682237798683df0512227d4120b21a4f5303cd5"} Nov 02 22:10:11.647978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:11.647907 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" event={"ID":"c8ab9390-8991-4fba-bd7e-d476fe1af0b4","Type":"ContainerStarted","Data":"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f"} Nov 02 22:10:11.648394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:11.648042 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:17.679573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:17.679538 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:10:17.795456 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:17.795400 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" podStartSLOduration=8.722005951 podStartE2EDuration="9.795384922s" totalImagesPullingTime="1.073378971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:11.801725421 +0000 UTC m=+3651.349975239" watchObservedRunningTime="2026-11-02 22:10:17.795384922 +0000 UTC m=+3657.343634703" Nov 02 22:10:25.947059 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:10:25.947023 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:10:31.505000 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.504968 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:10:31.509392 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.509364 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603369 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603324 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603562 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603445 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603562 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603508 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603675 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603564 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603707 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lv9q\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603769 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603727 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603769 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603753 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603850 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603805 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.603898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.603871 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705151 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705207 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705287 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705345 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705373 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705455 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705619 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lv9q\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705650 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.705730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.705689 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.706509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.706355 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.706598 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.706574 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.707025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.707003 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.707221 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.707198 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.708066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.708043 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.710398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.710373 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.711497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.711465 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.712603 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.712579 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:10:31.764698 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.764627 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.764895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.764870 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lv9q\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fwf6f\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:31.824386 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:31.824351 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:32.262397 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.262362 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:10:32.263307 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:10:32.263287 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8033aec2_77b5_433d_b47f_99679262bc56.slice/crio-3af51fe3fb45ac273b7544897d97d402d3d81ebcb760b95dbd2f1e18f8120558 WatchSource:0}: Error finding container 3af51fe3fb45ac273b7544897d97d402d3d81ebcb760b95dbd2f1e18f8120558: Status 404 returned error can't find the container with id 3af51fe3fb45ac273b7544897d97d402d3d81ebcb760b95dbd2f1e18f8120558 Nov 02 22:10:32.265917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.265882 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:32.266046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.266007 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:32.266115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.266056 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:32.862769 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.862731 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" event={"ID":"8033aec2-77b5-433d-b47f-99679262bc56","Type":"ContainerStarted","Data":"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd"} Nov 02 22:10:32.863238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:32.862798 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" event={"ID":"8033aec2-77b5-433d-b47f-99679262bc56","Type":"ContainerStarted","Data":"3af51fe3fb45ac273b7544897d97d402d3d81ebcb760b95dbd2f1e18f8120558"} Nov 02 22:10:33.081184 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:33.081123 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podStartSLOduration=2.081106332 podStartE2EDuration="2.081106332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.028544768 +0000 UTC m=+3672.576794566" watchObservedRunningTime="2026-11-02 22:10:33.081106332 +0000 UTC m=+3672.629356115" Nov 02 22:10:33.830222 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:33.830185 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:33.832044 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:33.832011 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:33.832214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:33.832077 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:34.827995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:34.827339 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:34.827995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:34.827401 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:35.825974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:35.825330 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:35.825974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:35.825395 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:36.826976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:36.826330 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:36.826976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:36.826406 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:37.827465 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:37.827418 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:37.828180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:37.828144 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:38.824978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:38.824922 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:38.825180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:38.825010 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:39.825417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:39.825373 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 02 22:10:39.825838 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:39.825446 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 02 22:10:41.310539 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:41.310502 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:41.311056 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:41.310825 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:41.311686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:41.311669 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:10:46.612279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.612244 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:10:46.616912 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.616892 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.678920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.678883 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679122 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.678931 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679122 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679057 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679122 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679083 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5nq\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679141 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679226 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679336 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679406 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.679463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.679444 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.723260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.723225 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:10:46.781074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781045 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781079 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781147 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781184 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781249 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781276 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5nq\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781332 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781437 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781507 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781537 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781723 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781754 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.781842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.781794 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.782212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.782185 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.783943 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.783921 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.784138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.784114 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.838695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.838662 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.838863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.838765 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5nq\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-njcp8\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:46.932604 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:46.932518 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:47.366263 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.366235 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:10:47.367881 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:10:47.367853 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b222d68_7e69_4776_ac81_97ae535841fb.slice/crio-60cd60f1c00d4eac970e20d808d0446a88d88251c9ffe4153326d36a9fc6104f WatchSource:0}: Error finding container 60cd60f1c00d4eac970e20d808d0446a88d88251c9ffe4153326d36a9fc6104f: Status 404 returned error can't find the container with id 60cd60f1c00d4eac970e20d808d0446a88d88251c9ffe4153326d36a9fc6104f Nov 02 22:10:47.370613 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.370577 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:47.370690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.370649 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:47.370690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.370679 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:47.988518 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.988482 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" event={"ID":"7b222d68-7e69-4776-ac81-97ae535841fb","Type":"ContainerStarted","Data":"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e"} Nov 02 22:10:47.988928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:47.988541 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" event={"ID":"7b222d68-7e69-4776-ac81-97ae535841fb","Type":"ContainerStarted","Data":"60cd60f1c00d4eac970e20d808d0446a88d88251c9ffe4153326d36a9fc6104f"} Nov 02 22:10:48.106465 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:48.106415 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podStartSLOduration=2.10640235 podStartE2EDuration="2.10640235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:48.103482111 +0000 UTC m=+3687.651731911" watchObservedRunningTime="2026-11-02 22:10:48.10640235 +0000 UTC m=+3687.654652143" Nov 02 22:10:48.933182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:48.933151 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:48.935014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:48.934973 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:48.935179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:48.935026 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:49.933461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:49.933423 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:49.933855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:49.933488 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:50.933333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:50.933297 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:50.933512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:50.933353 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:51.934049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:51.934000 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:51.934511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:51.934058 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:52.933307 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:52.933265 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:52.933510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:52.933329 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:53.933106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:53.933065 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:53.933514 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:53.933150 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:54.933050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:54.933008 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 02 22:10:54.933252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:54.933088 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 02 22:10:55.677925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.677882 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:10:55.682554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.682529 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.735173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.735132 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:10:55.735173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.735140 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:10:55.735173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.735133 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:10:55.735499 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.735242 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n4jrm\"" Nov 02 22:10:55.794623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.792520 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:10:55.865682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.865644 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.865912 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.865694 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.866042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.865912 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.866124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.866105 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.866411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.866232 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.866411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.866371 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvdz\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.866575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.866436 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.953795 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:10:55.953688 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:10:55.967625 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.967582 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.967821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.967634 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.967895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.967821 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.968020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.967997 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.968095 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.968048 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.968169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.968116 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvdz\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.968230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.968166 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.968856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.968816 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.970675 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.970642 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.970893 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.970866 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.971069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.971047 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:55.971424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:55.971401 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:56.027264 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.027206 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:56.027462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.027390 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvdz\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-67jnm\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:56.253441 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.253324 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:56.253837 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.253814 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:56.254596 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.254576 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:10:56.297427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.297397 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:56.738232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.738199 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:10:56.739113 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:10:56.739084 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc764f224_be3b_40aa_b1bb_4d4f8e5d6ea9.slice/crio-77fdf3415ea213c45bd8caa6bdffa67bbc50a32c37f72cb586ab0a22ba1aae2a WatchSource:0}: Error finding container 77fdf3415ea213c45bd8caa6bdffa67bbc50a32c37f72cb586ab0a22ba1aae2a: Status 404 returned error can't find the container with id 77fdf3415ea213c45bd8caa6bdffa67bbc50a32c37f72cb586ab0a22ba1aae2a Nov 02 22:10:56.741865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.741830 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:56.741993 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:56.741894 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:57.094039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.093991 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" event={"ID":"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9","Type":"ContainerStarted","Data":"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd"} Nov 02 22:10:57.094520 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.094067 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" event={"ID":"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9","Type":"ContainerStarted","Data":"77fdf3415ea213c45bd8caa6bdffa67bbc50a32c37f72cb586ab0a22ba1aae2a"} Nov 02 22:10:57.094520 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.094342 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:10:57.096228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.096201 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-67jnm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:10:57.096361 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.096261 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:10:57.210657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:57.210603 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" podStartSLOduration=2.210589528 podStartE2EDuration="2.210589528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:57.21020842 +0000 UTC m=+3696.758458220" watchObservedRunningTime="2026-11-02 22:10:57.210589528 +0000 UTC m=+3696.758839308" Nov 02 22:10:58.122536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:58.122501 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-67jnm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:10:58.123103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:10:58.122571 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:11:01.122828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:01.122793 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:11:14.039502 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:14.039410 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:11:14.039983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:14.039647 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" containerID="cri-o://c46780c0768213975b07af91c6f1998098ccea9c3c4a2f77125658175f1be7bb" gracePeriod=30 Nov 02 22:11:19.279211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.279171 5859 generic.go:356] "Generic (PLEG): container finished" podID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerID="c46780c0768213975b07af91c6f1998098ccea9c3c4a2f77125658175f1be7bb" exitCode=0 Nov 02 22:11:19.279657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.279249 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerDied","Data":"c46780c0768213975b07af91c6f1998098ccea9c3c4a2f77125658175f1be7bb"} Nov 02 22:11:19.279657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.279325 5859 scope.go:122] "RemoveContainer" containerID="338e7c832df6ccf603d4a4392406f7998418915fdb88d08f6b8cadda6325c09f" Nov 02 22:11:19.322440 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.322412 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:11:19.468224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468186 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468286 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468390 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468438 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468511 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468579 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468631 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468628 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:19.468793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468675 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468718 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\") pod \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\" (UID: \"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3\") " Nov 02 22:11:19.468918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.468852 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:19.469049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469024 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:19.469118 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469068 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:19.469198 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469152 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:19.469438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469402 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.469438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469424 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.469438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469437 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.469627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469452 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.469627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.469466 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.470833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.470810 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:11:19.470972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.470845 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "kube-api-access-rc2tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:19.471071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.471052 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:19.471129 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.471099 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token" pod "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" (UID: "250c73fa-bd6e-4c72-bae4-87e4c5ee11a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:19.570204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.570164 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.570204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.570196 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.570204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.570208 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2tv\" (UniqueName: \"kubernetes.io/projected/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-kube-api-access-rc2tv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:19.570441 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:19.570217 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:20.288102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.288069 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" Nov 02 22:11:20.288102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.288080 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2" event={"ID":"250c73fa-bd6e-4c72-bae4-87e4c5ee11a3","Type":"ContainerDied","Data":"69452f63912f6e04b0cddc737b3dfba7ab4a4c14cdcf9da4698c1789bbf1c964"} Nov 02 22:11:20.288758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.288146 5859 scope.go:122] "RemoveContainer" containerID="c46780c0768213975b07af91c6f1998098ccea9c3c4a2f77125658175f1be7bb" Nov 02 22:11:20.506452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.506416 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:11:20.556892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.556810 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8702a9-openshift-default-6bfd876dbf-742p2"] Nov 02 22:11:20.572078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:20.572046 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" path="/var/lib/kubelet/pods/250c73fa-bd6e-4c72-bae4-87e4c5ee11a3/volumes" Nov 02 22:11:24.151933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.151895 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:11:24.152784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.152149 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" podUID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" containerName="maas-api" containerID="cri-o://45157dad145e8d666d7de5b6aef028e0b3e6cd4804ad3be51a3fdd28c62a6953" gracePeriod=30 Nov 02 22:11:24.332530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.332494 5859 generic.go:356] "Generic (PLEG): container finished" podID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" containerID="45157dad145e8d666d7de5b6aef028e0b3e6cd4804ad3be51a3fdd28c62a6953" exitCode=0 Nov 02 22:11:24.333113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.332913 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" event={"ID":"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4","Type":"ContainerDied","Data":"45157dad145e8d666d7de5b6aef028e0b3e6cd4804ad3be51a3fdd28c62a6953"} Nov 02 22:11:24.441574 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.441546 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:11:24.535418 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.535381 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\") pod \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " Nov 02 22:11:24.535634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.535446 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\" (UniqueName: \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\") pod \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " Nov 02 22:11:24.535634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.535491 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\") pod \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " Nov 02 22:11:24.535634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.535549 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") pod \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\" (UID: \"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4\") " Nov 02 22:11:24.535855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.535830 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca" pod "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" (UID: "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:24.536198 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.536177 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:24.537753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.537723 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs" pod "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" (UID: "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:24.537753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.537746 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls" pod "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" (UID: "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:24.537896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.537749 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q" pod "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" (UID: "70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4"). InnerVolumeSpecName "kube-api-access-ggf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:24.637414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.637365 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggf6q\" (UniqueName: \"kubernetes.io/projected/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-kube-api-access-ggf6q\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:24.637414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.637410 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:24.637414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:24.637421 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:25.341270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.341236 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" Nov 02 22:11:25.341270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.341247 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v" event={"ID":"70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4","Type":"ContainerDied","Data":"26c3e23a7489c6c3ae08e7e4e87ae18f3ab255ce7f36be49c8dc28af443d064a"} Nov 02 22:11:25.341811 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.341314 5859 scope.go:122] "RemoveContainer" containerID="45157dad145e8d666d7de5b6aef028e0b3e6cd4804ad3be51a3fdd28c62a6953" Nov 02 22:11:25.447423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.447395 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:11:25.447607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.447588 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" podUID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" containerName="maas-api" containerID="cri-o://cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21" gracePeriod=30 Nov 02 22:11:25.511824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.511793 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:11:25.515935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.515908 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9981d2-589c754469-zvl4v"] Nov 02 22:11:25.724384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.724360 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:11:25.754727 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.754703 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\" (UniqueName: \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\") pod \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " Nov 02 22:11:25.754877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.754773 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\") pod \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " Nov 02 22:11:25.754966 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.754882 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") pod \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " Nov 02 22:11:25.754966 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.754938 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\") pod \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\" (UID: \"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03\") " Nov 02 22:11:25.755170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.755146 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca" pod "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" (UID: "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:25.755423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.755402 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:25.756994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.756924 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls" pod "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" (UID: "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:25.756994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.756943 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs" pod "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" (UID: "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:25.757124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.757095 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj" pod "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" (UID: "f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03"). InnerVolumeSpecName "kube-api-access-pzckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:25.856890 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.856862 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:25.856890 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.856887 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:25.857134 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:25.856896 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzckj\" (UniqueName: \"kubernetes.io/projected/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03-kube-api-access-pzckj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:25.945463 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:25.945377 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:11:26.349287 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.349258 5859 generic.go:356] "Generic (PLEG): container finished" podID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" containerID="cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21" exitCode=0 Nov 02 22:11:26.349741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.349306 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" Nov 02 22:11:26.349741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.349345 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" event={"ID":"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03","Type":"ContainerDied","Data":"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21"} Nov 02 22:11:26.349741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.349404 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs" event={"ID":"f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03","Type":"ContainerDied","Data":"e5880ffe9e4d357436cc3b3132131df0e3afa8667c3f530bc14bbbd4316f0121"} Nov 02 22:11:26.349741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.349432 5859 scope.go:122] "RemoveContainer" containerID="cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21" Nov 02 22:11:26.361146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.361126 5859 scope.go:122] "RemoveContainer" containerID="cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21" Nov 02 22:11:26.361432 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:26.361414 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21\": container with ID starting with cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21 not found: ID does not exist" containerID="cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21" Nov 02 22:11:26.361513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.361439 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21"} err="failed to get container status \"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21\": rpc error: code = NotFound desc = could not find container \"cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21\": container with ID starting with cbfaec1c68b469e65d7a3b14b2922a2ae27587d4a46cba9dc870844acf1add21 not found: ID does not exist" Nov 02 22:11:26.515971 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.515850 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:11:26.577004 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.576971 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" path="/var/lib/kubelet/pods/70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4/volumes" Nov 02 22:11:26.577382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:26.577350 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-6689da-766859b9d7-fksjs"] Nov 02 22:11:28.010762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:28.010733 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:11:28.011153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:28.010916 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" containerID="cri-o://e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9" gracePeriod=30 Nov 02 22:11:28.399626 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:28.399592 5859 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:11:28.399785 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:28.399649 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:11:28.571807 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:28.571773 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" path="/var/lib/kubelet/pods/f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03/volumes" Nov 02 22:11:33.283919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.283863 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:11:33.398267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398235 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398317 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398383 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398417 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398487 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398546 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398583 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398606 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:33.398789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398633 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398699 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\") pod \"b724c917-6f74-45a9-b517-bf5d039c678a\" (UID: \"b724c917-6f74-45a9-b517-bf5d039c678a\") " Nov 02 22:11:33.398898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398815 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:33.398898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:33.399117 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.398996 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:33.399344 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.399318 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.399447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.399346 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.399447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.399396 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b724c917-6f74-45a9-b517-bf5d039c678a-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.399447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.399409 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.399447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.399416 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:33.400868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.400665 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:11:33.400868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.400844 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "kube-api-access-xc6sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:33.401038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.400871 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:11:33.401038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.400935 5859 generic.go:356] "Generic (PLEG): container finished" podID="b724c917-6f74-45a9-b517-bf5d039c678a" containerID="e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9" exitCode=0 Nov 02 22:11:33.401038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.400998 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerDied","Data":"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9"} Nov 02 22:11:33.401038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.401013 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" Nov 02 22:11:33.401238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.401056 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9" event={"ID":"b724c917-6f74-45a9-b517-bf5d039c678a","Type":"ContainerDied","Data":"399ca3025deb3c8566a7110cdfff34b68648bbbb5141a6215723348373480c75"} Nov 02 22:11:33.401238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.401079 5859 scope.go:122] "RemoveContainer" containerID="e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9" Nov 02 22:11:33.401320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.401244 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token" pod "b724c917-6f74-45a9-b517-bf5d039c678a" (UID: "b724c917-6f74-45a9-b517-bf5d039c678a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:33.428014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.427993 5859 scope.go:122] "RemoveContainer" containerID="f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923" Nov 02 22:11:33.437327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.437297 5859 scope.go:122] "RemoveContainer" containerID="e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9" Nov 02 22:11:33.437577 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:33.437560 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9\": container with ID starting with e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9 not found: ID does not exist" containerID="e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9" Nov 02 22:11:33.437623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.437584 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9"} err="failed to get container status \"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9\": rpc error: code = NotFound desc = could not find container \"e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9\": container with ID starting with e7f82854915e9572dd42f7a8ac4760af5c517e2313fe42fad94710bf58a0bdd9 not found: ID does not exist" Nov 02 22:11:33.437623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.437600 5859 scope.go:122] "RemoveContainer" containerID="f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923" Nov 02 22:11:33.437823 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:33.437806 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923\": container with ID starting with f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923 not found: ID does not exist" containerID="f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923" Nov 02 22:11:33.437878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.437834 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923"} err="failed to get container status \"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923\": rpc error: code = NotFound desc = could not find container \"f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923\": container with ID starting with f3558a7aa827e5486097708844aa5a01327f919025583761d1e718d6b7f28923 not found: ID does not exist" Nov 02 22:11:33.500866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.500832 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.500866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.500859 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.500866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.500869 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6sd\" (UniqueName: \"kubernetes.io/projected/b724c917-6f74-45a9-b517-bf5d039c678a-kube-api-access-xc6sd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.501099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.500878 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b724c917-6f74-45a9-b517-bf5d039c678a-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.501099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.500887 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b724c917-6f74-45a9-b517-bf5d039c678a-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:33.823877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.823845 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:11:33.830720 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:33.830667 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6689da-openshift-default-8458d459d9-5rth9"] Nov 02 22:11:34.572467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:34.572437 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" path="/var/lib/kubelet/pods/b724c917-6f74-45a9-b517-bf5d039c678a/volumes" Nov 02 22:11:48.783151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:48.783118 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:11:53.785032 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:53.784991 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" podUID="b0c5765c-609b-4dfd-9438-869af50a639b" containerName="payload-pre-processing" containerID="cri-o://af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b" gracePeriod=25 Nov 02 22:11:54.044617 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.044564 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:11:54.133136 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.133110 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\" (UniqueName: \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\") pod \"b0c5765c-609b-4dfd-9438-869af50a639b\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " Nov 02 22:11:54.133352 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.133337 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\") pod \"b0c5765c-609b-4dfd-9438-869af50a639b\" (UID: \"b0c5765c-609b-4dfd-9438-869af50a639b\") " Nov 02 22:11:54.133738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.133715 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume" pod "b0c5765c-609b-4dfd-9438-869af50a639b" (UID: "b0c5765c-609b-4dfd-9438-869af50a639b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:54.135219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.135196 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd" pod "b0c5765c-609b-4dfd-9438-869af50a639b" (UID: "b0c5765c-609b-4dfd-9438-869af50a639b"). InnerVolumeSpecName "kube-api-access-4thvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:54.235404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.235378 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c5765c-609b-4dfd-9438-869af50a639b-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:54.235404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.235402 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thvd\" (UniqueName: \"kubernetes.io/projected/b0c5765c-609b-4dfd-9438-869af50a639b-kube-api-access-4thvd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:54.544913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.544877 5859 generic.go:356] "Generic (PLEG): container finished" podID="b0c5765c-609b-4dfd-9438-869af50a639b" containerID="af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b" exitCode=0 Nov 02 22:11:54.545113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.544929 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" Nov 02 22:11:54.545113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.544938 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" event={"ID":"b0c5765c-609b-4dfd-9438-869af50a639b","Type":"ContainerDied","Data":"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b"} Nov 02 22:11:54.545113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.545024 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n" event={"ID":"b0c5765c-609b-4dfd-9438-869af50a639b","Type":"ContainerDied","Data":"399a7bf4a67930a64aa946d88412359708e9c85a0694c9bf87938937a302d2e1"} Nov 02 22:11:54.545113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.545045 5859 scope.go:122] "RemoveContainer" containerID="af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b" Nov 02 22:11:54.555153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.555072 5859 scope.go:122] "RemoveContainer" containerID="af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b" Nov 02 22:11:54.555369 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:54.555348 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b\": container with ID starting with af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b not found: ID does not exist" containerID="af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b" Nov 02 22:11:54.555429 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.555382 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b"} err="failed to get container status \"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b\": rpc error: code = NotFound desc = could not find container \"af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b\": container with ID starting with af1fee60121403fd7f6ff8a46f9dd842fa7765b0af124bb5a83b21d5cf9bb44b not found: ID does not exist" Nov 02 22:11:54.709811 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.709775 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:11:54.758912 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:54.758880 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-363e15-698dcd6c5-szm4n"] Nov 02 22:11:55.981213 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:11:55.981174 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:11:56.570735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:11:56.570706 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c5765c-609b-4dfd-9438-869af50a639b" path="/var/lib/kubelet/pods/b0c5765c-609b-4dfd-9438-869af50a639b/volumes" Nov 02 22:12:25.944335 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:12:25.944296 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:12:55.945890 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:12:55.945851 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:13:07.299523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.299456 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:13:07.300226 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.299754 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" podUID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" containerName="maas-api" containerID="cri-o://11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f" gracePeriod=30 Nov 02 22:13:07.591488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.591462 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:13:07.722821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.722786 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\") pod \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " Nov 02 22:13:07.723012 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.722895 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\" (UniqueName: \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\") pod \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " Nov 02 22:13:07.723012 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.722993 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") pod \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " Nov 02 22:13:07.723130 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.723085 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\") pod \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\" (UID: \"c8ab9390-8991-4fba-bd7e-d476fe1af0b4\") " Nov 02 22:13:07.723343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.723314 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca" pod "c8ab9390-8991-4fba-bd7e-d476fe1af0b4" (UID: "c8ab9390-8991-4fba-bd7e-d476fe1af0b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:07.723677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.723612 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:07.725265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.725238 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs" pod "c8ab9390-8991-4fba-bd7e-d476fe1af0b4" (UID: "c8ab9390-8991-4fba-bd7e-d476fe1af0b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:07.725352 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.725280 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls" pod "c8ab9390-8991-4fba-bd7e-d476fe1af0b4" (UID: "c8ab9390-8991-4fba-bd7e-d476fe1af0b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:07.725566 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.725553 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl" pod "c8ab9390-8991-4fba-bd7e-d476fe1af0b4" (UID: "c8ab9390-8991-4fba-bd7e-d476fe1af0b4"). InnerVolumeSpecName "kube-api-access-hzdhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:07.825638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.825594 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:07.825638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.825637 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdhl\" (UniqueName: \"kubernetes.io/projected/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-kube-api-access-hzdhl\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:07.825931 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:07.825650 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8ab9390-8991-4fba-bd7e-d476fe1af0b4-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:08.028299 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.028262 5859 generic.go:356] "Generic (PLEG): container finished" podID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" containerID="11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f" exitCode=0 Nov 02 22:13:08.028477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.028306 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" Nov 02 22:13:08.028477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.028341 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" event={"ID":"c8ab9390-8991-4fba-bd7e-d476fe1af0b4","Type":"ContainerDied","Data":"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f"} Nov 02 22:13:08.028477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.028390 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht" event={"ID":"c8ab9390-8991-4fba-bd7e-d476fe1af0b4","Type":"ContainerDied","Data":"e9bd142ea2d704a10a3e43ee9682237798683df0512227d4120b21a4f5303cd5"} Nov 02 22:13:08.028477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.028411 5859 scope.go:122] "RemoveContainer" containerID="11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f" Nov 02 22:13:08.039396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.039378 5859 scope.go:122] "RemoveContainer" containerID="11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f" Nov 02 22:13:08.039664 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:13:08.039644 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f\": container with ID starting with 11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f not found: ID does not exist" containerID="11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f" Nov 02 22:13:08.039714 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.039673 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f"} err="failed to get container status \"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f\": rpc error: code = NotFound desc = could not find container \"11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f\": container with ID starting with 11104316f235885ddf51418de1226c4ef07e4de9b9d1c72207064c0ba16aae0f not found: ID does not exist" Nov 02 22:13:08.204175 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.204065 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:13:08.209606 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.209580 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-363e15-79f7487d79-wfqht"] Nov 02 22:13:08.571748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:13:08.571713 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" path="/var/lib/kubelet/pods/c8ab9390-8991-4fba-bd7e-d476fe1af0b4/volumes" Nov 02 22:13:25.945835 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:13:25.945794 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:13:55.944807 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:13:55.944768 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:14:25.945052 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:14:25.944974 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:14:37.011077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.011045 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:14:37.011650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.011253 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" podUID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" containerName="maas-api" containerID="cri-o://43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387" gracePeriod=30 Nov 02 22:14:37.297010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.296980 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:14:37.317046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.317020 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\") pod \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " Nov 02 22:14:37.317235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.317218 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\" (UniqueName: \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\") pod \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " Nov 02 22:14:37.317310 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.317252 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\") pod \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " Nov 02 22:14:37.317383 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.317318 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\") pod \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\" (UID: \"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55\") " Nov 02 22:14:37.317784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.317756 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca" pod "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" (UID: "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:37.319184 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.319156 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs" pod "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" (UID: "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:37.319758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.319736 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls" pod "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" (UID: "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:37.319829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.319793 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z" pod "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" (UID: "d023b45f-ebfd-45b0-b8f5-0b20aff0cf55"). InnerVolumeSpecName "kube-api-access-6jh6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:37.421789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.421754 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jh6z\" (UniqueName: \"kubernetes.io/projected/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-kube-api-access-6jh6z\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:37.422062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.422041 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:37.422208 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.422193 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:37.422312 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.422298 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:37.643087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.643053 5859 generic.go:356] "Generic (PLEG): container finished" podID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" containerID="43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387" exitCode=0 Nov 02 22:14:37.643289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.643101 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" Nov 02 22:14:37.643289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.643137 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" event={"ID":"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55","Type":"ContainerDied","Data":"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387"} Nov 02 22:14:37.643289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.643197 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk" event={"ID":"d023b45f-ebfd-45b0-b8f5-0b20aff0cf55","Type":"ContainerDied","Data":"01b008d3f43190a8f0161d584c5d9c0fa2919cc59ee6565dc2d70a1beab01482"} Nov 02 22:14:37.643289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.643228 5859 scope.go:122] "RemoveContainer" containerID="43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387" Nov 02 22:14:37.654548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.654531 5859 scope.go:122] "RemoveContainer" containerID="43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387" Nov 02 22:14:37.654811 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:14:37.654792 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387\": container with ID starting with 43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387 not found: ID does not exist" containerID="43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387" Nov 02 22:14:37.654860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.654819 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387"} err="failed to get container status \"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387\": rpc error: code = NotFound desc = could not find container \"43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387\": container with ID starting with 43b20a5d985dbf23c94b9c9721567e472a5f6166b588f56005828306b3077387 not found: ID does not exist" Nov 02 22:14:37.798075 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.798044 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:14:37.844218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:37.844195 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-5ddbd0-6dc5cd8945-qhcmk"] Nov 02 22:14:38.571331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:38.571290 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" path="/var/lib/kubelet/pods/d023b45f-ebfd-45b0-b8f5-0b20aff0cf55/volumes" Nov 02 22:14:40.940671 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.940639 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:14:40.941235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941221 5859 cpu_manager.go:414] "containerMap: removing container" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:14:40.941285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941235 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:14:40.941285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941246 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" containerName="maas-api" Nov 02 22:14:40.941285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941252 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" containerName="maas-api" Nov 02 22:14:40.941285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941272 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" containerName="maas-api" Nov 02 22:14:40.941285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941277 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941288 5859 cpu_manager.go:414] "containerMap: removing container" podUID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941293 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941305 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941309 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941339 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b0c5765c-609b-4dfd-9438-869af50a639b" containerName="payload-pre-processing" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941343 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c5765c-609b-4dfd-9438-869af50a639b" containerName="payload-pre-processing" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941357 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941364 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941381 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941386 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941488 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ab9390-8991-4fba-bd7e-d476fe1af0b4" containerName="maas-api" Nov 02 22:14:40.941501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941505 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941518 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c5765c-609b-4dfd-9438-869af50a639b" containerName="payload-pre-processing" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941531 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941540 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b724c917-6f74-45a9-b517-bf5d039c678a" containerName="istio-proxy" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941552 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941562 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d023b45f-ebfd-45b0-b8f5-0b20aff0cf55" containerName="maas-api" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941574 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d4ffcd-b3ac-42b8-a5bf-c0c0a76f3d03" containerName="maas-api" Nov 02 22:14:40.942092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.941585 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ee86d3-413e-4ad9-aaf0-bd3dc57fc9b4" containerName="maas-api" Nov 02 22:14:40.946690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.946671 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:40.995836 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:40.995812 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dd28e6-openshift-default-dockercfg-qbjtq\"" Nov 02 22:14:41.046165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.046139 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:14:41.071696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.071666 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.071841 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.071811 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.071965 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.071922 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072008 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072058 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072090 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072236 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072144 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x7w\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072423 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.072559 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.072538 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174306 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174271 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174329 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174414 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174464 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174502 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174544 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174568 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174642 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x7w\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174682 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174707 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.174764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174737 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.175005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.174979 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.175092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.175059 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.175282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.175259 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.176863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.176841 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.176988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.176971 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.230218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.230140 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.231337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.231306 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x7w\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\") pod \"e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.260526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.260502 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:41.691069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.691032 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:14:41.739411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.739369 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:14:41.739494 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.739457 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:14:41.739536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:41.739498 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:14:42.679743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:42.679705 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerStarted","Data":"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3"} Nov 02 22:14:42.680308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:42.679759 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerStarted","Data":"4f7638946d80b506af0d6e911adc2ebf9266e4cba93cfbc0ded38e0ddc8fad06"} Nov 02 22:14:42.883630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:42.883579 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podStartSLOduration=2.883567861 podStartE2EDuration="2.883567861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:42.835311276 +0000 UTC m=+3922.383561094" watchObservedRunningTime="2026-11-02 22:14:42.883567861 +0000 UTC m=+3922.431817641" Nov 02 22:14:43.261677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:43.261643 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:43.267199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:43.267176 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:43.685908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:43.685881 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:43.686683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:14:43.686667 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:14:55.944920 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:14:55.944885 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:15:00.195526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.195443 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:00.196525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.196492 5859 cpu_manager.go:414] "containerMap: removing container" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:15:00.196525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.196513 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250c73fa-bd6e-4c72-bae4-87e4c5ee11a3" containerName="istio-proxy" Nov 02 22:15:00.200599 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.200574 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:00.200808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.200789 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.204933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.204914 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.215106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.215085 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:00.219088 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.219069 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.241618 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.241593 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:00.245490 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.245473 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:00.245590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.245573 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.249545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.249521 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.291069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.291048 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.291168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.291104 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlns\" (UniqueName: \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.328390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.328363 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:15:00.364477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.364046 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:00.367734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.367711 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:00.392803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.392776 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.392930 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.392858 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.393022 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.392945 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp6m\" (UniqueName: \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.393067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393021 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.393130 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393114 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwv4\" (UniqueName: \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.393223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393203 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.393411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393248 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.393411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393378 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlns\" (UniqueName: \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.393521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393473 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmwx\" (UniqueName: \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.393578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.393548 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjmg\" (UniqueName: \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.394115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.394092 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.421613 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.421559 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:00.428121 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.428041 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:00.446691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.446628 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlns\" (UniqueName: \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.480064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.480034 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:00.494649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494624 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjmg\" (UniqueName: \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.494758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494672 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.494758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494710 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.494758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494731 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp6m\" (UniqueName: \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.494758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494748 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.495016 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494824 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwv4\" (UniqueName: \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.495016 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.494854 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.495137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.495018 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmwx\" (UniqueName: \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.495548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.495527 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.495643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.495556 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.495643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.495577 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.495720 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.495642 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.518596 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.518571 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:00.551547 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.551516 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwv4\" (UniqueName: \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\") pod \"maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.551681 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.551667 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjmg\" (UniqueName: \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.551798 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.551778 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp6m\" (UniqueName: \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.552401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.552378 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmwx\" (UniqueName: \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.561795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.561769 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:00.570156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.567184 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:00.824722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.824699 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:00.832762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.832741 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:00.895718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.891541 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:00.965535 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.965500 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:00.979024 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:15:00.978999 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6999be80_28f1_4ee8_9e0a_2969b29a7ab2.slice/crio-c641eb4b69766f9395d5f532bc648e70e58d56df753364dacf2810ad39a975c3 WatchSource:0}: Error finding container c641eb4b69766f9395d5f532bc648e70e58d56df753364dacf2810ad39a975c3: Status 404 returned error can't find the container with id c641eb4b69766f9395d5f532bc648e70e58d56df753364dacf2810ad39a975c3 Nov 02 22:15:00.993917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:00.993894 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:00.998034 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:15:00.997999 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f587bb_467d_4f38_a5b0_3acdd10de117.slice/crio-23358f6753068367f803922a285f4424f64d96ee15c068a3f113988236ee453d WatchSource:0}: Error finding container 23358f6753068367f803922a285f4424f64d96ee15c068a3f113988236ee453d: Status 404 returned error can't find the container with id 23358f6753068367f803922a285f4424f64d96ee15c068a3f113988236ee453d Nov 02 22:15:01.196917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.196887 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:01.208467 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:15:01.208438 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63fc8d9_5d56_44eb_a3a1_5645978b28db.slice/crio-b451791a0c51ff4e0a6e1f0cdc887f025e4cfb6fb794dbb8f4246d7198ddcd48 WatchSource:0}: Error finding container b451791a0c51ff4e0a6e1f0cdc887f025e4cfb6fb794dbb8f4246d7198ddcd48: Status 404 returned error can't find the container with id b451791a0c51ff4e0a6e1f0cdc887f025e4cfb6fb794dbb8f4246d7198ddcd48 Nov 02 22:15:01.208965 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.208931 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:01.216086 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:15:01.216051 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d169e96_7cdb_4dcd_ba12_7da50a8280f2.slice/crio-51d455b5b6cc76cb7c28f8330d08a0b272e67ec790b2c5a36eb38c1d90d848db WatchSource:0}: Error finding container 51d455b5b6cc76cb7c28f8330d08a0b272e67ec790b2c5a36eb38c1d90d848db: Status 404 returned error can't find the container with id 51d455b5b6cc76cb7c28f8330d08a0b272e67ec790b2c5a36eb38c1d90d848db Nov 02 22:15:01.801925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.801839 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerStarted","Data":"b451791a0c51ff4e0a6e1f0cdc887f025e4cfb6fb794dbb8f4246d7198ddcd48"} Nov 02 22:15:01.803620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.803583 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerStarted","Data":"fa3312183c11e2dbf14fd4d763d7f8dee1971819c7f0545a66f70ca2cc6ff2ac"} Nov 02 22:15:01.803746 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.803633 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerStarted","Data":"c641eb4b69766f9395d5f532bc648e70e58d56df753364dacf2810ad39a975c3"} Nov 02 22:15:01.805230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.805192 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerStarted","Data":"c7ff1be74215a03a06510b92e1d91c59ce2522ef0853463fdfd1e8c6f6aa91b9"} Nov 02 22:15:01.805850 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.805373 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerStarted","Data":"23358f6753068367f803922a285f4424f64d96ee15c068a3f113988236ee453d"} Nov 02 22:15:01.807265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.807053 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerStarted","Data":"66913e4b8d8719c941b4f55f5c4b8f4b1e4625256c7dfef536a53e4535d06773"} Nov 02 22:15:01.807371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.807356 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerStarted","Data":"0b88c84af186113c0ea1cd0e5764af13c54ef7a9dc73a469fc3f2c03d550125e"} Nov 02 22:15:01.808736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.808704 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerStarted","Data":"51d455b5b6cc76cb7c28f8330d08a0b272e67ec790b2c5a36eb38c1d90d848db"} Nov 02 22:15:01.911878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:01.911812 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" podStartSLOduration=1.911793266 podStartE2EDuration="1.911793266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:01.909065216 +0000 UTC m=+3941.457314998" watchObservedRunningTime="2026-11-02 22:15:01.911793266 +0000 UTC m=+3941.460043050" Nov 02 22:15:02.015544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.015499 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" podStartSLOduration=2.015484716 podStartE2EDuration="2.015484716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.013273912 +0000 UTC m=+3941.561523692" watchObservedRunningTime="2026-11-02 22:15:02.015484716 +0000 UTC m=+3941.563734499" Nov 02 22:15:02.128815 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.128759 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" podStartSLOduration=2.128742967 podStartE2EDuration="2.128742967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:02.125675547 +0000 UTC m=+3941.673925328" watchObservedRunningTime="2026-11-02 22:15:02.128742967 +0000 UTC m=+3941.676992748" Nov 02 22:15:02.816766 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.816733 5859 generic.go:356] "Generic (PLEG): container finished" podID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerID="cfabf483b74d0c318eb2dec9f90b538a277fd12d1d40e12b9f5fa590c9282bd8" exitCode=60 Nov 02 22:15:02.817254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.816818 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerDied","Data":"cfabf483b74d0c318eb2dec9f90b538a277fd12d1d40e12b9f5fa590c9282bd8"} Nov 02 22:15:02.817254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.817072 5859 scope.go:122] "RemoveContainer" containerID="cfabf483b74d0c318eb2dec9f90b538a277fd12d1d40e12b9f5fa590c9282bd8" Nov 02 22:15:02.818643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.818580 5859 generic.go:356] "Generic (PLEG): container finished" podID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerID="fa3312183c11e2dbf14fd4d763d7f8dee1971819c7f0545a66f70ca2cc6ff2ac" exitCode=60 Nov 02 22:15:02.818716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.818658 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerDied","Data":"fa3312183c11e2dbf14fd4d763d7f8dee1971819c7f0545a66f70ca2cc6ff2ac"} Nov 02 22:15:02.818874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.818857 5859 scope.go:122] "RemoveContainer" containerID="fa3312183c11e2dbf14fd4d763d7f8dee1971819c7f0545a66f70ca2cc6ff2ac" Nov 02 22:15:02.820433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.820419 5859 generic.go:356] "Generic (PLEG): container finished" podID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerID="c7ff1be74215a03a06510b92e1d91c59ce2522ef0853463fdfd1e8c6f6aa91b9" exitCode=60 Nov 02 22:15:02.820509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.820488 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerDied","Data":"c7ff1be74215a03a06510b92e1d91c59ce2522ef0853463fdfd1e8c6f6aa91b9"} Nov 02 22:15:02.820705 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.820675 5859 scope.go:122] "RemoveContainer" containerID="c7ff1be74215a03a06510b92e1d91c59ce2522ef0853463fdfd1e8c6f6aa91b9" Nov 02 22:15:02.822467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.822451 5859 generic.go:356] "Generic (PLEG): container finished" podID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerID="66913e4b8d8719c941b4f55f5c4b8f4b1e4625256c7dfef536a53e4535d06773" exitCode=60 Nov 02 22:15:02.822546 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.822514 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerDied","Data":"66913e4b8d8719c941b4f55f5c4b8f4b1e4625256c7dfef536a53e4535d06773"} Nov 02 22:15:02.822756 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.822739 5859 scope.go:122] "RemoveContainer" containerID="66913e4b8d8719c941b4f55f5c4b8f4b1e4625256c7dfef536a53e4535d06773" Nov 02 22:15:02.824146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.824121 5859 generic.go:356] "Generic (PLEG): container finished" podID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerID="0d0529b06290a30949ff5430a3a57e052612527b4f022184e21f13b95c45d487" exitCode=60 Nov 02 22:15:02.824263 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.824203 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerDied","Data":"0d0529b06290a30949ff5430a3a57e052612527b4f022184e21f13b95c45d487"} Nov 02 22:15:02.824357 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:02.824343 5859 scope.go:122] "RemoveContainer" containerID="0d0529b06290a30949ff5430a3a57e052612527b4f022184e21f13b95c45d487" Nov 02 22:15:03.833662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.833639 5859 generic.go:356] "Generic (PLEG): container finished" podID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerID="978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079" exitCode=60 Nov 02 22:15:03.833999 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.833723 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerDied","Data":"978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079"} Nov 02 22:15:03.833999 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.833796 5859 scope.go:122] "RemoveContainer" containerID="cfabf483b74d0c318eb2dec9f90b538a277fd12d1d40e12b9f5fa590c9282bd8" Nov 02 22:15:03.834117 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.834071 5859 scope.go:122] "RemoveContainer" containerID="978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079" Nov 02 22:15:03.834369 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:03.834346 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b_odh-ai-gateway-infra(b63fc8d9-5d56-44eb-a3a1-5645978b28db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" Nov 02 22:15:03.836007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.835941 5859 generic.go:356] "Generic (PLEG): container finished" podID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerID="9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56" exitCode=60 Nov 02 22:15:03.836007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.835992 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerDied","Data":"9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56"} Nov 02 22:15:03.836292 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.836274 5859 scope.go:122] "RemoveContainer" containerID="9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56" Nov 02 22:15:03.836548 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:03.836526 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w_odh-ai-gateway-infra(6999be80-28f1-4ee8-9e0a-2969b29a7ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" Nov 02 22:15:03.838000 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.837978 5859 generic.go:356] "Generic (PLEG): container finished" podID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerID="bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1" exitCode=60 Nov 02 22:15:03.838103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.838055 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerDied","Data":"bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1"} Nov 02 22:15:03.838345 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.838327 5859 scope.go:122] "RemoveContainer" containerID="bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1" Nov 02 22:15:03.838566 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:03.838548 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4_odh-ai-gateway-infra(11f587bb-467d-4f38-a5b0-3acdd10de117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" Nov 02 22:15:03.839932 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.839910 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerStarted","Data":"688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a"} Nov 02 22:15:03.841628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.841608 5859 generic.go:356] "Generic (PLEG): container finished" podID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerID="628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249" exitCode=60 Nov 02 22:15:03.841718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.841688 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerDied","Data":"628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249"} Nov 02 22:15:03.841859 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.841846 5859 scope.go:122] "RemoveContainer" containerID="628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249" Nov 02 22:15:03.842083 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:03.842065 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg_odh-ai-gateway-infra(6d169e96-7cdb-4dcd-ba12-7da50a8280f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" Nov 02 22:15:03.925984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.925934 5859 scope.go:122] "RemoveContainer" containerID="fa3312183c11e2dbf14fd4d763d7f8dee1971819c7f0545a66f70ca2cc6ff2ac" Nov 02 22:15:03.938768 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.938748 5859 scope.go:122] "RemoveContainer" containerID="c7ff1be74215a03a06510b92e1d91c59ce2522ef0853463fdfd1e8c6f6aa91b9" Nov 02 22:15:03.953063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:03.953044 5859 scope.go:122] "RemoveContainer" containerID="0d0529b06290a30949ff5430a3a57e052612527b4f022184e21f13b95c45d487" Nov 02 22:15:04.854921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.854888 5859 scope.go:122] "RemoveContainer" containerID="bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1" Nov 02 22:15:04.855375 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:04.855233 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4_odh-ai-gateway-infra(11f587bb-467d-4f38-a5b0-3acdd10de117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" Nov 02 22:15:04.857330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.857303 5859 generic.go:356] "Generic (PLEG): container finished" podID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerID="688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a" exitCode=60 Nov 02 22:15:04.857460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.857339 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerDied","Data":"688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a"} Nov 02 22:15:04.857460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.857389 5859 scope.go:122] "RemoveContainer" containerID="66913e4b8d8719c941b4f55f5c4b8f4b1e4625256c7dfef536a53e4535d06773" Nov 02 22:15:04.857633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.857617 5859 scope.go:122] "RemoveContainer" containerID="688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a" Nov 02 22:15:04.857909 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:04.857886 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j_odh-ai-gateway-infra(3bc63eb1-5507-431c-a5cc-5ed4528de54b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" Nov 02 22:15:04.859386 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.859365 5859 scope.go:122] "RemoveContainer" containerID="628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249" Nov 02 22:15:04.859613 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:04.859588 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg_odh-ai-gateway-infra(6d169e96-7cdb-4dcd-ba12-7da50a8280f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" Nov 02 22:15:04.861367 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.861335 5859 scope.go:122] "RemoveContainer" containerID="978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079" Nov 02 22:15:04.861552 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:04.861530 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b_odh-ai-gateway-infra(b63fc8d9-5d56-44eb-a3a1-5645978b28db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" Nov 02 22:15:04.862854 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:04.862836 5859 scope.go:122] "RemoveContainer" containerID="9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56" Nov 02 22:15:04.863123 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:04.863104 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w_odh-ai-gateway-infra(6999be80-28f1-4ee8-9e0a-2969b29a7ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" Nov 02 22:15:05.871502 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:05.871474 5859 scope.go:122] "RemoveContainer" containerID="688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a" Nov 02 22:15:05.871928 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:05.871780 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j_odh-ai-gateway-infra(3bc63eb1-5507-431c-a5cc-5ed4528de54b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" Nov 02 22:15:14.563665 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:14.563634 5859 scope.go:122] "RemoveContainer" containerID="978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079" Nov 02 22:15:14.564251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:14.564060 5859 scope.go:122] "RemoveContainer" containerID="9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56" Nov 02 22:15:14.930807 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:14.930780 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerStarted","Data":"8ac75340fc933fdbb88a1243a5b33049bd19b1b94f84abc5d904c6be7d625631"} Nov 02 22:15:14.932601 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:14.932577 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerStarted","Data":"40c36a581e85a562f934b718ba7037af7d2b37365875324cc9d809901b5b70e2"} Nov 02 22:15:15.076230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.076193 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" podStartSLOduration=15.076181278 podStartE2EDuration="15.076181278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:15.074165327 +0000 UTC m=+3954.622415127" watchObservedRunningTime="2026-11-02 22:15:15.076181278 +0000 UTC m=+3954.624431058" Nov 02 22:15:15.564234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.564199 5859 scope.go:122] "RemoveContainer" containerID="bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1" Nov 02 22:15:15.564618 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.564267 5859 scope.go:122] "RemoveContainer" containerID="628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249" Nov 02 22:15:15.940824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.940796 5859 generic.go:356] "Generic (PLEG): container finished" podID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerID="8ac75340fc933fdbb88a1243a5b33049bd19b1b94f84abc5d904c6be7d625631" exitCode=60 Nov 02 22:15:15.940935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.940866 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerDied","Data":"8ac75340fc933fdbb88a1243a5b33049bd19b1b94f84abc5d904c6be7d625631"} Nov 02 22:15:15.941019 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.940975 5859 scope.go:122] "RemoveContainer" containerID="978a95142a8b7ef8a0f9ad22137c409c47ec1e82c68d297a07c5ce8fa0c8f079" Nov 02 22:15:15.941290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.941270 5859 scope.go:122] "RemoveContainer" containerID="8ac75340fc933fdbb88a1243a5b33049bd19b1b94f84abc5d904c6be7d625631" Nov 02 22:15:15.941579 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:15.941551 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b_odh-ai-gateway-infra(b63fc8d9-5d56-44eb-a3a1-5645978b28db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" Nov 02 22:15:15.942833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.942813 5859 generic.go:356] "Generic (PLEG): container finished" podID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerID="40c36a581e85a562f934b718ba7037af7d2b37365875324cc9d809901b5b70e2" exitCode=60 Nov 02 22:15:15.942918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.942881 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerDied","Data":"40c36a581e85a562f934b718ba7037af7d2b37365875324cc9d809901b5b70e2"} Nov 02 22:15:15.943232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.943213 5859 scope.go:122] "RemoveContainer" containerID="40c36a581e85a562f934b718ba7037af7d2b37365875324cc9d809901b5b70e2" Nov 02 22:15:15.943499 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:15.943453 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w_odh-ai-gateway-infra(6999be80-28f1-4ee8-9e0a-2969b29a7ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" Nov 02 22:15:15.944695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.944678 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerStarted","Data":"89b317504e7be347ba0ca5fa4e46485452adab1da8742b550aa0ff662528d3e7"} Nov 02 22:15:15.952938 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:15.952919 5859 scope.go:122] "RemoveContainer" containerID="9ec97d887b337d543b8f639b95bc1a5a8667c4f5e2286a0b12f666943009fe56" Nov 02 22:15:16.085712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.085637 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:16.156002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.155936 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" podStartSLOduration=16.155923012 podStartE2EDuration="16.155923012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:16.152921495 +0000 UTC m=+3955.701171293" watchObservedRunningTime="2026-11-02 22:15:16.155923012 +0000 UTC m=+3955.704172792" Nov 02 22:15:16.193561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.193527 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:16.955968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.955921 5859 generic.go:356] "Generic (PLEG): container finished" podID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerID="b74ec31936f8c4a9ef5602e31a603a2375fcf159d14dab6ed3dc1376302edbf3" exitCode=60 Nov 02 22:15:16.956294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.955981 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerDied","Data":"b74ec31936f8c4a9ef5602e31a603a2375fcf159d14dab6ed3dc1376302edbf3"} Nov 02 22:15:16.956294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.956037 5859 scope.go:122] "RemoveContainer" containerID="bf594bbc881ab735e9809cb2454f25bf6ba29880c3c05d3614924ff897b4d7c1" Nov 02 22:15:16.956397 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.956380 5859 scope.go:122] "RemoveContainer" containerID="b74ec31936f8c4a9ef5602e31a603a2375fcf159d14dab6ed3dc1376302edbf3" Nov 02 22:15:16.956706 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:16.956684 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4_odh-ai-gateway-infra(11f587bb-467d-4f38-a5b0-3acdd10de117)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" Nov 02 22:15:16.958829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.958808 5859 generic.go:356] "Generic (PLEG): container finished" podID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerID="89b317504e7be347ba0ca5fa4e46485452adab1da8742b550aa0ff662528d3e7" exitCode=60 Nov 02 22:15:16.958923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.958834 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerDied","Data":"89b317504e7be347ba0ca5fa4e46485452adab1da8742b550aa0ff662528d3e7"} Nov 02 22:15:16.959161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.959147 5859 scope.go:122] "RemoveContainer" containerID="89b317504e7be347ba0ca5fa4e46485452adab1da8742b550aa0ff662528d3e7" Nov 02 22:15:16.959398 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:16.959380 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg_odh-ai-gateway-infra(6d169e96-7cdb-4dcd-ba12-7da50a8280f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" Nov 02 22:15:16.969821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:16.969798 5859 scope.go:122] "RemoveContainer" containerID="628c64daa2edfdcebc51f3807c0d006861e4111dafb26ea9c134f28ee00a8249" Nov 02 22:15:17.119989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.119936 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:17.155578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.155555 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:17.164359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.164332 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:17.192969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.192935 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\") pod \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " Nov 02 22:15:17.193251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.193231 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\" (UniqueName: \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\") pod \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\" (UID: \"b63fc8d9-5d56-44eb-a3a1-5645978b28db\") " Nov 02 22:15:17.193320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.193301 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca" pod "b63fc8d9-5d56-44eb-a3a1-5645978b28db" (UID: "b63fc8d9-5d56-44eb-a3a1-5645978b28db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:17.193667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.193648 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63fc8d9-5d56-44eb-a3a1-5645978b28db-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.195468 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.195441 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx" pod "b63fc8d9-5d56-44eb-a3a1-5645978b28db" (UID: "b63fc8d9-5d56-44eb-a3a1-5645978b28db"). InnerVolumeSpecName "kube-api-access-tcmwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:17.294289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.294267 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\") pod \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " Nov 02 22:15:17.294459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.294447 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\" (UniqueName: \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\") pod \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\" (UID: \"6999be80-28f1-4ee8-9e0a-2969b29a7ab2\") " Nov 02 22:15:17.294579 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.294560 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca" pod "6999be80-28f1-4ee8-9e0a-2969b29a7ab2" (UID: "6999be80-28f1-4ee8-9e0a-2969b29a7ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:17.294833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.294818 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.294894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.294835 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmwx\" (UniqueName: \"kubernetes.io/projected/b63fc8d9-5d56-44eb-a3a1-5645978b28db-kube-api-access-tcmwx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.296508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.296480 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg" pod "6999be80-28f1-4ee8-9e0a-2969b29a7ab2" (UID: "6999be80-28f1-4ee8-9e0a-2969b29a7ab2"). InnerVolumeSpecName "kube-api-access-hvjmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:17.396356 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.396332 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjmg\" (UniqueName: \"kubernetes.io/projected/6999be80-28f1-4ee8-9e0a-2969b29a7ab2-kube-api-access-hvjmg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:17.564368 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.564300 5859 scope.go:122] "RemoveContainer" containerID="688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a" Nov 02 22:15:17.966410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.966384 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" Nov 02 22:15:17.966410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.966400 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b" event={"ID":"b63fc8d9-5d56-44eb-a3a1-5645978b28db","Type":"ContainerDied","Data":"b451791a0c51ff4e0a6e1f0cdc887f025e4cfb6fb794dbb8f4246d7198ddcd48"} Nov 02 22:15:17.966852 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.966462 5859 scope.go:122] "RemoveContainer" containerID="8ac75340fc933fdbb88a1243a5b33049bd19b1b94f84abc5d904c6be7d625631" Nov 02 22:15:17.967793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.967767 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" Nov 02 22:15:17.967885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.967806 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w" event={"ID":"6999be80-28f1-4ee8-9e0a-2969b29a7ab2","Type":"ContainerDied","Data":"c641eb4b69766f9395d5f532bc648e70e58d56df753364dacf2810ad39a975c3"} Nov 02 22:15:17.972059 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.972038 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerStarted","Data":"290e8581ddb316a8b86c7ddca1b282739a7b7459c1dca6f5d991f030bb10ea33"} Nov 02 22:15:17.978458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:17.978437 5859 scope.go:122] "RemoveContainer" containerID="40c36a581e85a562f934b718ba7037af7d2b37365875324cc9d809901b5b70e2" Nov 02 22:15:18.141380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.141355 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:18.205903 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.205878 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\" (UniqueName: \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\") pod \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " Nov 02 22:15:18.206072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.206051 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\") pod \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\" (UID: \"6d169e96-7cdb-4dcd-ba12-7da50a8280f2\") " Nov 02 22:15:18.206378 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.206357 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca" pod "6d169e96-7cdb-4dcd-ba12-7da50a8280f2" (UID: "6d169e96-7cdb-4dcd-ba12-7da50a8280f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:18.206591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.206575 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.207989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.207962 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4" pod "6d169e96-7cdb-4dcd-ba12-7da50a8280f2" (UID: "6d169e96-7cdb-4dcd-ba12-7da50a8280f2"). InnerVolumeSpecName "kube-api-access-wcwv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:18.308001 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.307973 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwv4\" (UniqueName: \"kubernetes.io/projected/6d169e96-7cdb-4dcd-ba12-7da50a8280f2-kube-api-access-wcwv4\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.330534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.330510 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:18.334640 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.334613 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7cde84-29894295-vpd4b"] Nov 02 22:15:18.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.399900 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:18.435062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.435034 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:18.443670 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.443644 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-58ca7f-29894295-nlg8w"] Nov 02 22:15:18.539114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.539092 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:18.575225 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.575192 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" path="/var/lib/kubelet/pods/6999be80-28f1-4ee8-9e0a-2969b29a7ab2/volumes" Nov 02 22:15:18.575859 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.575832 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" path="/var/lib/kubelet/pods/b63fc8d9-5d56-44eb-a3a1-5645978b28db/volumes" Nov 02 22:15:18.610961 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.610932 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\" (UniqueName: \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\") pod \"11f587bb-467d-4f38-a5b0-3acdd10de117\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " Nov 02 22:15:18.611078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.611038 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\") pod \"11f587bb-467d-4f38-a5b0-3acdd10de117\" (UID: \"11f587bb-467d-4f38-a5b0-3acdd10de117\") " Nov 02 22:15:18.611387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.611363 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca" pod "11f587bb-467d-4f38-a5b0-3acdd10de117" (UID: "11f587bb-467d-4f38-a5b0-3acdd10de117"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:18.611517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.611503 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f587bb-467d-4f38-a5b0-3acdd10de117-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.612831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.612808 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m" pod "11f587bb-467d-4f38-a5b0-3acdd10de117" (UID: "11f587bb-467d-4f38-a5b0-3acdd10de117"). InnerVolumeSpecName "kube-api-access-nnp6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:18.712994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.712917 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnp6m\" (UniqueName: \"kubernetes.io/projected/11f587bb-467d-4f38-a5b0-3acdd10de117-kube-api-access-nnp6m\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:18.984256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.984178 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" Nov 02 22:15:18.984256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.984188 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4" event={"ID":"11f587bb-467d-4f38-a5b0-3acdd10de117","Type":"ContainerDied","Data":"23358f6753068367f803922a285f4424f64d96ee15c068a3f113988236ee453d"} Nov 02 22:15:18.984776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.984276 5859 scope.go:122] "RemoveContainer" containerID="b74ec31936f8c4a9ef5602e31a603a2375fcf159d14dab6ed3dc1376302edbf3" Nov 02 22:15:18.986359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.986334 5859 generic.go:356] "Generic (PLEG): container finished" podID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerID="290e8581ddb316a8b86c7ddca1b282739a7b7459c1dca6f5d991f030bb10ea33" exitCode=60 Nov 02 22:15:18.986492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.986405 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerDied","Data":"290e8581ddb316a8b86c7ddca1b282739a7b7459c1dca6f5d991f030bb10ea33"} Nov 02 22:15:18.986813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.986793 5859 scope.go:122] "RemoveContainer" containerID="290e8581ddb316a8b86c7ddca1b282739a7b7459c1dca6f5d991f030bb10ea33" Nov 02 22:15:18.987102 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:18.987081 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j_odh-ai-gateway-infra(3bc63eb1-5507-431c-a5cc-5ed4528de54b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" Nov 02 22:15:18.987856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.987838 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" event={"ID":"6d169e96-7cdb-4dcd-ba12-7da50a8280f2","Type":"ContainerDied","Data":"51d455b5b6cc76cb7c28f8330d08a0b272e67ec790b2c5a36eb38c1d90d848db"} Nov 02 22:15:18.988000 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.987842 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg" Nov 02 22:15:18.999545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:18.999521 5859 scope.go:122] "RemoveContainer" containerID="688f1b811a7ee130349f9292774fe47a6f5f6dfd9d89c93e52a54a3c40d50a9a" Nov 02 22:15:19.013670 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.013653 5859 scope.go:122] "RemoveContainer" containerID="89b317504e7be347ba0ca5fa4e46485452adab1da8742b550aa0ff662528d3e7" Nov 02 22:15:19.134212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.134182 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:19.145004 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.144978 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:19.151685 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.151661 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894295-rptx4"] Nov 02 22:15:19.354033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.353991 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:19.357977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:19.357936 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4585ee-29894295-dkhxg"] Nov 02 22:15:20.165358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.165334 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:20.233177 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.233138 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\" (UniqueName: \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\") pod \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " Nov 02 22:15:20.233393 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.233314 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\") pod \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\" (UID: \"3bc63eb1-5507-431c-a5cc-5ed4528de54b\") " Nov 02 22:15:20.233676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.233654 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca" pod "3bc63eb1-5507-431c-a5cc-5ed4528de54b" (UID: "3bc63eb1-5507-431c-a5cc-5ed4528de54b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.235455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.235429 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns" pod "3bc63eb1-5507-431c-a5cc-5ed4528de54b" (UID: "3bc63eb1-5507-431c-a5cc-5ed4528de54b"). InnerVolumeSpecName "kube-api-access-krlns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.335443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.335418 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlns\" (UniqueName: \"kubernetes.io/projected/3bc63eb1-5507-431c-a5cc-5ed4528de54b-kube-api-access-krlns\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.335443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.335444 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc63eb1-5507-431c-a5cc-5ed4528de54b-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.576042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.576012 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" path="/var/lib/kubelet/pods/11f587bb-467d-4f38-a5b0-3acdd10de117/volumes" Nov 02 22:15:20.576561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:20.576542 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" path="/var/lib/kubelet/pods/6d169e96-7cdb-4dcd-ba12-7da50a8280f2/volumes" Nov 02 22:15:21.012015 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:21.011984 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" Nov 02 22:15:21.012235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:21.011986 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j" event={"ID":"3bc63eb1-5507-431c-a5cc-5ed4528de54b","Type":"ContainerDied","Data":"0b88c84af186113c0ea1cd0e5764af13c54ef7a9dc73a469fc3f2c03d550125e"} Nov 02 22:15:21.012235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:21.012146 5859 scope.go:122] "RemoveContainer" containerID="290e8581ddb316a8b86c7ddca1b282739a7b7459c1dca6f5d991f030bb10ea33" Nov 02 22:15:21.166058 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:21.166029 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:21.168869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:21.168841 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06e461-29894295-g4n4j"] Nov 02 22:15:22.571343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:15:22.571310 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" path="/var/lib/kubelet/pods/3bc63eb1-5507-431c-a5cc-5ed4528de54b/volumes" Nov 02 22:15:25.943384 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:25.943351 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:15:55.946060 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:15:55.946019 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:16:25.945719 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:16:25.945681 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:16:28.036572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.036526 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:16:28.037331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.036804 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" containerID="cri-o://651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd" gracePeriod=30 Nov 02 22:16:28.121870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.121839 5859 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-67jnm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:8080/ready\": dial tcp 10.134.0.172:8080: connect: connection refused" start-of-body= Nov 02 22:16:28.122070 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.121898 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.172:8080/ready\": dial tcp 10.134.0.172:8080: connect: connection refused" Nov 02 22:16:28.301346 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.301281 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:16:28.372194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372157 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372236 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372260 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372302 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372372 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372579 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372474 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372581 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\") pod \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\" (UID: \"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9\") " Nov 02 22:16:28.372747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.372726 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:28.373240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.373197 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-ca-configmap\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.374777 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.374738 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:28.374913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.374772 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:28.374913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.374893 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:28.375047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.375022 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:28.375110 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.375094 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "kube-api-access-ltvdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:28.375200 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.375181 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert" pod "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" (UID: "c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:28.474804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474771 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvdz\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-kube-api-access-ltvdz\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.474804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474800 5859 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-local-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.474804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474810 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.475074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474820 5859 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-kubeconfig\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.475074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474829 5859 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-cacerts\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.475074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.474837 5859 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9-istio-csr-dns-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:28.506761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.506729 5859 generic.go:356] "Generic (PLEG): container finished" podID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerID="651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd" exitCode=0 Nov 02 22:16:28.506935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.506772 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" Nov 02 22:16:28.506935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.506798 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" event={"ID":"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9","Type":"ContainerDied","Data":"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd"} Nov 02 22:16:28.506935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.506848 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm" event={"ID":"c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9","Type":"ContainerDied","Data":"77fdf3415ea213c45bd8caa6bdffa67bbc50a32c37f72cb586ab0a22ba1aae2a"} Nov 02 22:16:28.506935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.506869 5859 scope.go:122] "RemoveContainer" containerID="651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd" Nov 02 22:16:28.518653 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.518625 5859 scope.go:122] "RemoveContainer" containerID="651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd" Nov 02 22:16:28.518939 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:16:28.518921 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd\": container with ID starting with 651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd not found: ID does not exist" containerID="651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd" Nov 02 22:16:28.519028 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.518965 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd"} err="failed to get container status \"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd\": rpc error: code = NotFound desc = could not find container \"651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd\": container with ID starting with 651147654e719f2bbb256019d074a9eb7bc31694dc1d524b7313c4d4ab298cdd not found: ID does not exist" Nov 02 22:16:28.661545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.661503 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:16:28.666759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:28.666727 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-67jnm"] Nov 02 22:16:30.572349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:30.572317 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" path="/var/lib/kubelet/pods/c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9/volumes" Nov 02 22:16:38.590973 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.590930 5859 generic.go:356] "Generic (PLEG): container finished" podID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerID="59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3" exitCode=0 Nov 02 22:16:38.591390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.590979 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerDied","Data":"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3"} Nov 02 22:16:38.591390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.591364 5859 scope.go:122] "RemoveContainer" containerID="59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3" Nov 02 22:16:38.593006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.592981 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:38.593085 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.593047 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:38.593123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:38.593079 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:39.599280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:39.599250 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerStarted","Data":"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154"} Nov 02 22:16:40.261152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:40.261120 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:16:40.262164 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:40.262138 5859 patch_prober.go:29] interesting pod/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:16:40.262293 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:40.262188 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 02 22:16:41.261418 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:41.261383 5859 patch_prober.go:29] interesting pod/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:16:41.261832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:41.261438 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 02 22:16:42.261471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:42.261440 5859 patch_prober.go:29] interesting pod/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:16:42.261852 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:42.261494 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 02 22:16:43.261284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:43.261249 5859 patch_prober.go:29] interesting pod/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:16:43.261459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:43.261302 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 02 22:16:43.686255 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:43.686225 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:16:44.261568 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:44.261527 5859 patch_prober.go:29] interesting pod/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 02 22:16:44.261783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:44.261597 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 02 22:16:45.266326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:45.266298 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:16:45.267451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:16:45.267428 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:16:55.946487 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:16:55.946448 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:17:25.944581 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:17:25.944546 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:17:40.221510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.221477 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:17:40.222191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222174 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" Nov 02 22:17:40.222191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222191 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222201 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222206 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222219 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222224 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222236 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222241 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222253 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222258 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222265 5859 cpu_manager.go:414] "containerMap: removing container" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222270 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222278 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222283 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222296 5859 cpu_manager.go:414] "containerMap: removing container" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222301 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222310 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222315 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222325 5859 cpu_manager.go:414] "containerMap: removing container" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222329 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222335 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222339 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222350 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.222375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222354 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222437 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222445 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222451 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222457 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c764f224-be3b-40aa-b1bb-4d4f8e5d6ea9" containerName="discovery" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222464 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222469 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222478 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222487 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222494 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222502 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222507 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222512 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222517 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f587bb-467d-4f38-a5b0-3acdd10de117" containerName="cleanup" Nov 02 22:17:40.223140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.222522 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:40.225988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.225972 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.309180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.306125 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:17:40.342567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342531 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.342755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342582 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.342755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342621 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jc9\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.342755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342669 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.342755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342693 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.342755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342748 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.343053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342807 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.343053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342836 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.343053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.342974 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.354024 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.353998 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cf2a53-openshift-default-dockercfg-bgjh7\"" Nov 02 22:17:40.444018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.443945 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444016 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444045 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jc9\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444081 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444106 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444170 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444229 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444255 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444367 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444668 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444765 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444716 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444815 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444789 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444810 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.444926 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.444911 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.446749 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.446730 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.446934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.446918 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.497558 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.497488 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.497699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.497676 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jc9\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\") pod \"e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.541843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.541821 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:40.965501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:40.965472 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:17:41.010139 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:17:41.010112 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30ab5d3_cc73_4fa0_957a_67da0b498a2b.slice/crio-a3fd951f2a7a1fb43f13608671dbdb8d9873e80670eb353ba8075d8baf713b39 WatchSource:0}: Error finding container a3fd951f2a7a1fb43f13608671dbdb8d9873e80670eb353ba8075d8baf713b39: Status 404 returned error can't find the container with id a3fd951f2a7a1fb43f13608671dbdb8d9873e80670eb353ba8075d8baf713b39 Nov 02 22:17:41.012530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:41.012500 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:41.012618 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:41.012566 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:41.012618 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:41.012612 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:41.045345 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:41.045319 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerStarted","Data":"a3fd951f2a7a1fb43f13608671dbdb8d9873e80670eb353ba8075d8baf713b39"} Nov 02 22:17:42.054331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:42.054295 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerStarted","Data":"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd"} Nov 02 22:17:42.209045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:42.208989 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podStartSLOduration=2.208973722 podStartE2EDuration="2.208973722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:42.208641527 +0000 UTC m=+4101.756891332" watchObservedRunningTime="2026-11-02 22:17:42.208973722 +0000 UTC m=+4101.757223503" Nov 02 22:17:42.541999 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:42.541943 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:42.547525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:42.547504 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:43.061119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:43.061088 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:43.061857 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:43.061838 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:17:48.602503 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.602462 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:17:48.603590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603551 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:48.603590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603577 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603630 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603638 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6999be80-28f1-4ee8-9e0a-2969b29a7ab2" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603662 5859 cpu_manager.go:414] "containerMap: removing container" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603670 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d169e96-7cdb-4dcd-ba12-7da50a8280f2" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603699 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603708 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:48.603865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603866 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc63eb1-5507-431c-a5cc-5ed4528de54b" containerName="cleanup" Nov 02 22:17:48.604475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.603882 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63fc8d9-5d56-44eb-a3a1-5645978b28db" containerName="cleanup" Nov 02 22:17:48.609031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.609012 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.655517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.655482 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:17:48.698559 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.698531 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf2a53\"" Nov 02 22:17:48.698559 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.698550 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf2a53-dockercfg-gm5dv\"" Nov 02 22:17:48.759928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.759901 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.760073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.759997 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k5h\" (UniqueName: \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.861712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.861653 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.861712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.861686 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k5h\" (UniqueName: \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.862393 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.862375 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.914711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.914679 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k5h\" (UniqueName: \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\") pod \"payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:48.924783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:48.924762 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:49.291314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:49.291239 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:17:49.296215 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:17:49.296178 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c0cf1d_ed63_4049_be16_b959ff929931.slice/crio-82811ca9668a3be41662038b975b6b0e48b3336c274aed00837f49dba315b7fc WatchSource:0}: Error finding container 82811ca9668a3be41662038b975b6b0e48b3336c274aed00837f49dba315b7fc: Status 404 returned error can't find the container with id 82811ca9668a3be41662038b975b6b0e48b3336c274aed00837f49dba315b7fc Nov 02 22:17:50.108832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:50.108795 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" event={"ID":"f9c0cf1d-ed63-4049-be16-b959ff929931","Type":"ContainerStarted","Data":"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91"} Nov 02 22:17:50.109231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:50.108853 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:17:50.109231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:50.108871 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" event={"ID":"f9c0cf1d-ed63-4049-be16-b959ff929931","Type":"ContainerStarted","Data":"82811ca9668a3be41662038b975b6b0e48b3336c274aed00837f49dba315b7fc"} Nov 02 22:17:50.215688 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:50.215642 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" podStartSLOduration=2.215628446 podStartE2EDuration="2.215628446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:50.213588343 +0000 UTC m=+4109.761838124" watchObservedRunningTime="2026-11-02 22:17:50.215628446 +0000 UTC m=+4109.763878276" Nov 02 22:17:52.307321 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.307289 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:17:52.311747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.311727 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.412339 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-db69f7-openshift-default-dockercfg-64l62\"" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.412912 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413039 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413068 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413118 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpbq\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413270 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413307 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413358 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413384 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.419285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.413415 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.448749 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.448711 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:17:52.515205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515165 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515222 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515281 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515307 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515337 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515402 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515495 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515518 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515574 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpbq\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515621 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515671 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515725 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.515983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.515905 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.516661 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.516620 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.519937 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.519915 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.520234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.520213 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.571929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.571895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.572147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.572125 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpbq\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\") pod \"e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:52.627458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:52.627435 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:53.080387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:53.080348 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:17:53.086113 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:17:53.086086 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8395503e_dab6_4946_be2f_f8d4698ddb72.slice/crio-16f75d02180ad26a4d1ca699e919f84ad39899c97d507c9bab33fe01d756f309 WatchSource:0}: Error finding container 16f75d02180ad26a4d1ca699e919f84ad39899c97d507c9bab33fe01d756f309: Status 404 returned error can't find the container with id 16f75d02180ad26a4d1ca699e919f84ad39899c97d507c9bab33fe01d756f309 Nov 02 22:17:53.089257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:53.089215 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:53.089372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:53.089303 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:53.089372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:53.089342 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:53.133219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:53.132872 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerStarted","Data":"16f75d02180ad26a4d1ca699e919f84ad39899c97d507c9bab33fe01d756f309"} Nov 02 22:17:54.141235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:54.141206 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerStarted","Data":"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942"} Nov 02 22:17:54.346265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:54.346214 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podStartSLOduration=2.346200999 podStartE2EDuration="2.346200999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:54.298807185 +0000 UTC m=+4113.847056994" watchObservedRunningTime="2026-11-02 22:17:54.346200999 +0000 UTC m=+4113.894450780" Nov 02 22:17:54.627993 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:54.627939 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:54.633199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:54.633175 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:55.147632 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:55.147601 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:55.148509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:17:55.148479 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:17:55.945102 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:17:55.945059 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:18:01.118861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:18:01.118834 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:18:25.944351 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:18:25.944313 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:18:55.946735 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:18:55.946696 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:19:20.393832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.393802 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:19:20.398470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.398447 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.447994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.447946 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-db69f7-serving-cert\"" Nov 02 22:19:20.452147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.452123 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:19:20.481830 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.481801 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.481984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.481871 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bv4\" (UniqueName: \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.482078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.482053 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.482143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.482110 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.583531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.583506 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bv4\" (UniqueName: \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.583696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.583592 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.583696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.583623 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.583817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.583775 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.583920 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:19:20.583901 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-serving-cert: secret "maas-api-e2e-inf-a-db69f7-serving-cert" not found Nov 02 22:19:20.583986 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:19:20.583977 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls podName:3d2dbd68-06a6-4370-a58e-888d0e0761c3 nodeName:}" failed. No retries permitted until 2026-11-02 22:19:21.083962823 +0000 UTC m=+4200.632212612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls") pod "maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" (UID: "3d2dbd68-06a6-4370-a58e-888d0e0761c3") : secret "maas-api-e2e-inf-a-db69f7-serving-cert" not found Nov 02 22:19:20.584296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.584279 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.585875 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.585858 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:20.637278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:20.637258 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bv4\" (UniqueName: \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:21.098848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.098811 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:21.101323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.101290 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") pod \"maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:21.312299 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.312272 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:21.687869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.687795 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:19:21.692699 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:19:21.692674 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2dbd68_06a6_4370_a58e_888d0e0761c3.slice/crio-1aa6a8376eecc0adaaba7216df00eaf62b377bd775f6cd588525cf255c254d2e WatchSource:0}: Error finding container 1aa6a8376eecc0adaaba7216df00eaf62b377bd775f6cd588525cf255c254d2e: Status 404 returned error can't find the container with id 1aa6a8376eecc0adaaba7216df00eaf62b377bd775f6cd588525cf255c254d2e Nov 02 22:19:21.694435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.694418 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:19:21.783483 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:21.783456 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" event={"ID":"3d2dbd68-06a6-4370-a58e-888d0e0761c3","Type":"ContainerStarted","Data":"1aa6a8376eecc0adaaba7216df00eaf62b377bd775f6cd588525cf255c254d2e"} Nov 02 22:19:22.792550 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:22.792520 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" event={"ID":"3d2dbd68-06a6-4370-a58e-888d0e0761c3","Type":"ContainerStarted","Data":"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c"} Nov 02 22:19:22.792944 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:22.792634 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:22.902431 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:22.902386 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" podStartSLOduration=2.036742561 podStartE2EDuration="2.902372676s" totalImagesPullingTime="865.630115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:22.899280346 +0000 UTC m=+4202.447530149" watchObservedRunningTime="2026-11-02 22:19:22.902372676 +0000 UTC m=+4202.450622472" Nov 02 22:19:25.945223 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:19:25.945187 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:19:28.806135 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:28.806108 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:19:48.004865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:48.004820 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:48.334031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:48.318856 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:55.946799 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:19:55.946755 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:19:56.191398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:56.189093 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:19:56.191398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:56.189235 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:57.073307 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.073277 5859 generic.go:356] "Generic (PLEG): container finished" podID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerID="aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942" exitCode=0 Nov 02 22:19:57.073715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.073358 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerDied","Data":"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942"} Nov 02 22:19:57.073715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.073689 5859 scope.go:122] "RemoveContainer" containerID="aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942" Nov 02 22:19:57.075059 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.075031 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:19:57.075154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.075097 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:19:57.075154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:57.075128 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:19:58.081464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:58.081431 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerStarted","Data":"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e"} Nov 02 22:19:58.630173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:58.630135 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:19:58.630375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:58.630310 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 02 22:19:58.630375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:58.630363 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 02 22:19:59.629395 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:59.629356 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 02 22:19:59.630056 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:19:59.629425 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 02 22:20:00.628974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:00.628290 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 02 22:20:00.628974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:00.628356 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 02 22:20:01.630325 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:01.630291 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 02 22:20:01.630806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:01.630351 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 02 22:20:02.630317 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:02.630284 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 02 22:20:02.630492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:02.630332 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 02 22:20:03.632505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:03.632474 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:20:03.632899 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:03.632860 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:20:03.633662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:03.633638 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:20:25.944680 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:20:25.944648 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:20:40.397556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.397527 5859 generic.go:356] "Generic (PLEG): container finished" podID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerID="c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd" exitCode=0 Nov 02 22:20:40.397940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.397596 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerDied","Data":"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd"} Nov 02 22:20:40.397940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.397923 5859 scope.go:122] "RemoveContainer" containerID="c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd" Nov 02 22:20:40.399459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.399435 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:40.399523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.399501 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:40.399561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:40.399550 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:41.431193 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:41.431152 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerStarted","Data":"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f"} Nov 02 22:20:41.542033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:41.541980 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:20:41.543484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:41.543456 5859 patch_prober.go:29] interesting pod/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 22:20:41.543616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:41.543514 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 22:20:42.542612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:42.542571 5859 patch_prober.go:29] interesting pod/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 22:20:42.543097 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:42.543059 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 22:20:43.062042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:43.062010 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:20:43.542538 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:43.542502 5859 patch_prober.go:29] interesting pod/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 22:20:43.542708 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:43.542559 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 22:20:44.542388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:44.542352 5859 patch_prober.go:29] interesting pod/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 22:20:44.542576 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:44.542411 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 22:20:45.542506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:45.542472 5859 patch_prober.go:29] interesting pod/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 22:20:45.542904 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:45.542517 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 22:20:46.546999 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:46.546965 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:20:46.548152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:20:46.548128 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:20:55.949739 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:20:55.949697 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:21:11.282101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.282064 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:21:11.285964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.285934 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.402347 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.402310 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b5e8794c-dockercfg-p5lb8\"" Nov 02 22:21:11.412497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.402687 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b5e8794c\"" Nov 02 22:21:11.416240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.416214 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:21:11.504639 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.504606 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.504908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.504869 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27f7\" (UniqueName: \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.606402 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.606364 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27f7\" (UniqueName: \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.606656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.606637 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.607300 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.607283 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.659082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.659059 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27f7\" (UniqueName: \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\") pod \"payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:11.900839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:11.900767 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:12.271796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:12.271645 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:21:12.275991 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:21:12.275966 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7689b141_cf35_4fe8_8803_da9250b08bc8.slice/crio-3c930f1c5ac85aa7a5bcbe595949e9b3dcfb3dd31204ac899d0ec03d6f16559e WatchSource:0}: Error finding container 3c930f1c5ac85aa7a5bcbe595949e9b3dcfb3dd31204ac899d0ec03d6f16559e: Status 404 returned error can't find the container with id 3c930f1c5ac85aa7a5bcbe595949e9b3dcfb3dd31204ac899d0ec03d6f16559e Nov 02 22:21:12.690180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:12.690149 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" event={"ID":"7689b141-cf35-4fe8-8803-da9250b08bc8","Type":"ContainerStarted","Data":"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9"} Nov 02 22:21:12.690597 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:12.690201 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" event={"ID":"7689b141-cf35-4fe8-8803-da9250b08bc8","Type":"ContainerStarted","Data":"3c930f1c5ac85aa7a5bcbe595949e9b3dcfb3dd31204ac899d0ec03d6f16559e"} Nov 02 22:21:12.690597 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:12.690257 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:12.797071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:12.797024 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" podStartSLOduration=1.797010744 podStartE2EDuration="1.797010744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:12.794666661 +0000 UTC m=+4312.342916468" watchObservedRunningTime="2026-11-02 22:21:12.797010744 +0000 UTC m=+4312.345260521" Nov 02 22:21:23.699095 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:21:23.699064 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:21:25.945321 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:21:25.945285 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:21:55.945214 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:21:55.945181 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:22:06.759207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.759164 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:22:06.763801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.763780 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.829322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.829286 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"openshift-service-ca.crt\"" Nov 02 22:22:06.834760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.834730 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:22:06.853771 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.853729 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.853977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.853870 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.853977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.853899 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.854072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.854019 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.854168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.854143 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985kt\" (UniqueName: \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.854276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.854258 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.860077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.860055 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"kube-root-ca.crt\"" Nov 02 22:22:06.860194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.860104 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"default-dockercfg-44ldk\"" Nov 02 22:22:06.860194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.860182 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7cde84\"/\"sub-test-model-7cde84-kserve-self-signed-certs\"" Nov 02 22:22:06.955196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955150 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955269 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985kt\" (UniqueName: \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955350 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955467 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955537 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955565 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.955984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955947 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.956079 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.955940 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.957540 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.957515 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.958291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.958269 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:06.958454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:06.958435 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:07.008340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:07.008310 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-985kt\" (UniqueName: \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\") pod \"sub-test-model-7cde84-kserve-6678458df5-p9rz2\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:07.077170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:07.077137 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:07.450570 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:07.450470 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:22:07.455192 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:22:07.455162 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5924f8f_bece_4802_b555_b7164ca81fd6.slice/crio-801286d1d3184175aac7789c105fbefe5374fb177f978c01373ccc2e770cb883 WatchSource:0}: Error finding container 801286d1d3184175aac7789c105fbefe5374fb177f978c01373ccc2e770cb883: Status 404 returned error can't find the container with id 801286d1d3184175aac7789c105fbefe5374fb177f978c01373ccc2e770cb883 Nov 02 22:22:08.071495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:08.071450 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" event={"ID":"c5924f8f-bece-4802-b555-b7164ca81fd6","Type":"ContainerStarted","Data":"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241"} Nov 02 22:22:08.071495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:08.071501 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" event={"ID":"c5924f8f-bece-4802-b555-b7164ca81fd6","Type":"ContainerStarted","Data":"801286d1d3184175aac7789c105fbefe5374fb177f978c01373ccc2e770cb883"} Nov 02 22:22:08.077854 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:08.077819 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:08.179983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:08.179915 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" podStartSLOduration=2.179902114 podStartE2EDuration="2.179902114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:08.177083907 +0000 UTC m=+4367.725333729" watchObservedRunningTime="2026-11-02 22:22:08.179902114 +0000 UTC m=+4367.728151938" Nov 02 22:22:17.077581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:17.077543 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:17.090369 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:17.090340 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:17.145303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:17.145270 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:22:25.944928 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:22:25.944887 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:22:27.802067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.802032 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:22:27.806241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.806222 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:27.902662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.902633 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3edc8c-serving-cert\"" Nov 02 22:22:27.905835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.905815 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:22:27.922228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.922203 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:27.922303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.922273 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:27.922381 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.922366 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbntm\" (UniqueName: \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:27.922502 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:27.922481 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.023530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.023499 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbntm\" (UniqueName: \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.023684 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.023611 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.023684 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.023667 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.023773 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:22:28.023700 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-serving-cert: secret "maas-api-e2e-inf-b-3edc8c-serving-cert" not found Nov 02 22:22:28.023773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.023740 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.023848 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:22:28.023787 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls podName:5513733d-cdf4-4a34-a804-afdf280ba870 nodeName:}" failed. No retries permitted until 2026-11-02 22:22:28.523767367 +0000 UTC m=+4388.072017136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls") pod "maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" (UID: "5513733d-cdf4-4a34-a804-afdf280ba870") : secret "maas-api-e2e-inf-b-3edc8c-serving-cert" not found Nov 02 22:22:28.024322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.024302 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.026108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.026089 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.078419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.078390 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbntm\" (UniqueName: \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.533448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.533363 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.535811 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.535790 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") pod \"maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:28.719686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:28.719657 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:29.089397 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:29.089365 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:22:29.093775 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:22:29.093746 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5513733d_cdf4_4a34_a804_afdf280ba870.slice/crio-58b5088aa035d3e1cc3a5e81e1a4070c83e6604beeb1860d2eb83c3a536743bb WatchSource:0}: Error finding container 58b5088aa035d3e1cc3a5e81e1a4070c83e6604beeb1860d2eb83c3a536743bb: Status 404 returned error can't find the container with id 58b5088aa035d3e1cc3a5e81e1a4070c83e6604beeb1860d2eb83c3a536743bb Nov 02 22:22:29.224538 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:29.224507 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" event={"ID":"5513733d-cdf4-4a34-a804-afdf280ba870","Type":"ContainerStarted","Data":"58b5088aa035d3e1cc3a5e81e1a4070c83e6604beeb1860d2eb83c3a536743bb"} Nov 02 22:22:30.232387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:30.232358 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" event={"ID":"5513733d-cdf4-4a34-a804-afdf280ba870","Type":"ContainerStarted","Data":"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e"} Nov 02 22:22:30.232745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:30.232486 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:30.387097 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:30.387044 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" podStartSLOduration=2.430126833 podStartE2EDuration="3.38702965s" totalImagesPullingTime="956.902817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:30.384489416 +0000 UTC m=+4389.932739219" watchObservedRunningTime="2026-11-02 22:22:30.38702965 +0000 UTC m=+4389.935279431" Nov 02 22:22:36.243967 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:36.243921 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:22:48.821907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.821836 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:22:48.826178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.826160 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.922170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.922138 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"kube-root-ca.crt\"" Nov 02 22:22:48.922438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.922228 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"openshift-service-ca.crt\"" Nov 02 22:22:48.922580 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.922483 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"default-dockercfg-ptbhj\"" Nov 02 22:22:48.922580 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.922490 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-58ca7f\"/\"sub-test-model-58ca7f-kserve-self-signed-certs\"" Nov 02 22:22:48.925430 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.925405 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:22:48.987477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.987433 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtf5\" (UniqueName: \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.987871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.987843 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.988041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.987911 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.988041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.988026 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.988152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.988073 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:48.988209 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:48.988194 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.088943 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089040 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089095 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089244 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtf5\" (UniqueName: \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089284 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089305 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089335 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089491 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.089576 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.089538 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.091419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.091389 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.091573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.091556 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.144293 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.144270 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtf5\" (UniqueName: \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\") pod \"sub-test-model-58ca7f-kserve-b6794d97c-2ssdp\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.438179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.438099 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:49.807783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:49.807706 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:22:49.812489 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:22:49.812465 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22092d4e_5eec_42e8_89e5_99175541f327.slice/crio-4702532be801661eaf61d2b3795717195d9cb1f79d4c4296bf8337df07f0c50e WatchSource:0}: Error finding container 4702532be801661eaf61d2b3795717195d9cb1f79d4c4296bf8337df07f0c50e: Status 404 returned error can't find the container with id 4702532be801661eaf61d2b3795717195d9cb1f79d4c4296bf8337df07f0c50e Nov 02 22:22:50.373273 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:50.373237 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" event={"ID":"22092d4e-5eec-42e8-89e5-99175541f327","Type":"ContainerStarted","Data":"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f"} Nov 02 22:22:50.373662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:50.373282 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" event={"ID":"22092d4e-5eec-42e8-89e5-99175541f327","Type":"ContainerStarted","Data":"4702532be801661eaf61d2b3795717195d9cb1f79d4c4296bf8337df07f0c50e"} Nov 02 22:22:50.438774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:50.438750 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:50.481793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:50.481747 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" podStartSLOduration=2.481733701 podStartE2EDuration="2.481733701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:50.47956027 +0000 UTC m=+4410.027810049" watchObservedRunningTime="2026-11-02 22:22:50.481733701 +0000 UTC m=+4410.029983482" Nov 02 22:22:55.948061 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:22:55.947988 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:22:59.438310 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:59.438282 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:22:59.450371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:22:59.450349 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:23:00.497138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:23:00.497104 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:23:25.945235 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:23:25.945153 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:23:55.945104 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:23:55.945062 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:24:17.551441 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:17.551404 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:24:22.552676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.552634 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" podUID="7689b141-cf35-4fe8-8803-da9250b08bc8" containerName="payload-pre-processing" containerID="cri-o://52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9" gracePeriod=25 Nov 02 22:24:22.825863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.825828 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:24:22.897739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.897714 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\") pod \"7689b141-cf35-4fe8-8803-da9250b08bc8\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " Nov 02 22:24:22.897887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.897782 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\" (UniqueName: \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\") pod \"7689b141-cf35-4fe8-8803-da9250b08bc8\" (UID: \"7689b141-cf35-4fe8-8803-da9250b08bc8\") " Nov 02 22:24:22.898179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.898155 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume" pod "7689b141-cf35-4fe8-8803-da9250b08bc8" (UID: "7689b141-cf35-4fe8-8803-da9250b08bc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:22.899815 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.899797 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7" pod "7689b141-cf35-4fe8-8803-da9250b08bc8" (UID: "7689b141-cf35-4fe8-8803-da9250b08bc8"). InnerVolumeSpecName "kube-api-access-q27f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:22.999547 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.999519 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7689b141-cf35-4fe8-8803-da9250b08bc8-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:22.999547 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:22.999547 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27f7\" (UniqueName: \"kubernetes.io/projected/7689b141-cf35-4fe8-8803-da9250b08bc8-kube-api-access-q27f7\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:23.067055 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.067027 5859 generic.go:356] "Generic (PLEG): container finished" podID="7689b141-cf35-4fe8-8803-da9250b08bc8" containerID="52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9" exitCode=0 Nov 02 22:24:23.067179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.067078 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" Nov 02 22:24:23.067179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.067111 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" event={"ID":"7689b141-cf35-4fe8-8803-da9250b08bc8","Type":"ContainerDied","Data":"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9"} Nov 02 22:24:23.067249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.067184 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb" event={"ID":"7689b141-cf35-4fe8-8803-da9250b08bc8","Type":"ContainerDied","Data":"3c930f1c5ac85aa7a5bcbe595949e9b3dcfb3dd31204ac899d0ec03d6f16559e"} Nov 02 22:24:23.067249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.067205 5859 scope.go:122] "RemoveContainer" containerID="52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9" Nov 02 22:24:23.078427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.078412 5859 scope.go:122] "RemoveContainer" containerID="52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9" Nov 02 22:24:23.078688 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:24:23.078667 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9\": container with ID starting with 52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9 not found: ID does not exist" containerID="52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9" Nov 02 22:24:23.078749 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.078694 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9"} err="failed to get container status \"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9\": rpc error: code = NotFound desc = could not find container \"52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9\": container with ID starting with 52c4cdee0987af63ea89853481050e32666e3869db98bae8677d35a76d2658d9 not found: ID does not exist" Nov 02 22:24:23.220663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.220637 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:24:23.224656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:23.224637 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b5e8794c-db785c575-nxjxb"] Nov 02 22:24:24.572279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:24:24.572250 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7689b141-cf35-4fe8-8803-da9250b08bc8" path="/var/lib/kubelet/pods/7689b141-cf35-4fe8-8803-da9250b08bc8/volumes" Nov 02 22:24:25.943531 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:24:25.943500 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:24:55.946532 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:24:55.946489 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:25:22.762666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.762633 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:25:22.763089 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.762823 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" containerID="cri-o://bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c" gracePeriod=30 Nov 02 22:25:22.808877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.808837 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:25:22.809082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.809048 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" containerID="cri-o://870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88" gracePeriod=30 Nov 02 22:25:22.811828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.811803 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:25:22.812094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.812004 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" containerID="cri-o://720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e" gracePeriod=30 Nov 02 22:25:22.855253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.854979 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:25:22.855253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:22.855150 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" containerID="cri-o://52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd" gracePeriod=30 Nov 02 22:25:23.439101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.439069 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:25:23.439302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.439260 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" podUID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" containerName="maas-api" containerID="cri-o://388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c" gracePeriod=30 Nov 02 22:25:23.730531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.730505 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:25:23.746737 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.746714 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\") pod \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " Nov 02 22:25:23.746870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.746767 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\" (UniqueName: \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\") pod \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " Nov 02 22:25:23.746933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.746912 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\") pod \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " Nov 02 22:25:23.747069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.747055 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") pod \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\" (UID: \"3d2dbd68-06a6-4370-a58e-888d0e0761c3\") " Nov 02 22:25:23.747132 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.747080 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca" pod "3d2dbd68-06a6-4370-a58e-888d0e0761c3" (UID: "3d2dbd68-06a6-4370-a58e-888d0e0761c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:23.747739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.747718 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d2dbd68-06a6-4370-a58e-888d0e0761c3-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:23.748946 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.748919 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4" pod "3d2dbd68-06a6-4370-a58e-888d0e0761c3" (UID: "3d2dbd68-06a6-4370-a58e-888d0e0761c3"). InnerVolumeSpecName "kube-api-access-t5bv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:23.749065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.749036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs" pod "3d2dbd68-06a6-4370-a58e-888d0e0761c3" (UID: "3d2dbd68-06a6-4370-a58e-888d0e0761c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:23.749065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.749056 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls" pod "3d2dbd68-06a6-4370-a58e-888d0e0761c3" (UID: "3d2dbd68-06a6-4370-a58e-888d0e0761c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:23.850045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.850008 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:23.853116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.852989 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d2dbd68-06a6-4370-a58e-888d0e0761c3-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:23.853116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:23.853023 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bv4\" (UniqueName: \"kubernetes.io/projected/3d2dbd68-06a6-4370-a58e-888d0e0761c3-kube-api-access-t5bv4\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:24.489974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.489918 5859 generic.go:356] "Generic (PLEG): container finished" podID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" containerID="388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c" exitCode=0 Nov 02 22:25:24.490196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.489989 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" event={"ID":"3d2dbd68-06a6-4370-a58e-888d0e0761c3","Type":"ContainerDied","Data":"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c"} Nov 02 22:25:24.490196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.490022 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" Nov 02 22:25:24.490196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.490046 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7" event={"ID":"3d2dbd68-06a6-4370-a58e-888d0e0761c3","Type":"ContainerDied","Data":"1aa6a8376eecc0adaaba7216df00eaf62b377bd775f6cd588525cf255c254d2e"} Nov 02 22:25:24.490196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.490071 5859 scope.go:122] "RemoveContainer" containerID="388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c" Nov 02 22:25:24.502197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.502181 5859 scope.go:122] "RemoveContainer" containerID="388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c" Nov 02 22:25:24.502450 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:24.502434 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c\": container with ID starting with 388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c not found: ID does not exist" containerID="388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c" Nov 02 22:25:24.502491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.502457 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c"} err="failed to get container status \"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c\": rpc error: code = NotFound desc = could not find container \"388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c\": container with ID starting with 388c3c9ae6a4d4944308aef79d55e27783de0412f5e9226fdcca2127b61fe14c not found: ID does not exist" Nov 02 22:25:24.643679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.643648 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:25:24.689979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:24.689934 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db69f7-85664f989d-2ndk7"] Nov 02 22:25:25.711363 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:25.711323 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-z8bpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:25.711808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:25.711392 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:25.738170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:25.738137 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rpfpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:25.738320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:25.738200 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:25.944103 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:25.944063 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:25:26.300266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:26.300229 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-njcp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:26.301469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:26.301425 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:26.332091 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:26.332058 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fwf6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:26.332450 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:26.332412 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:26.572759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:26.572712 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" path="/var/lib/kubelet/pods/3d2dbd68-06a6-4370-a58e-888d0e0761c3/volumes" Nov 02 22:25:28.082377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.082353 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144250 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144341 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144387 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144451 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144571 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144601 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144646 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144688 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.145895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.144803 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\") pod \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\" (UID: \"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23\") " Nov 02 22:25:28.149589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.149497 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.149778 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.149758 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.149917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.149891 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.150158 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.149981 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.150238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.150169 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.150238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.150225 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.150387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.150367 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.151690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.151661 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "kube-api-access-mxckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.152914 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.152888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy" pod "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" (UID: "da5a4e23-2b6f-48e6-970b-1eacc5b5cf23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.162436 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.162416 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:25:28.203190 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.203166 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:25:28.208592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.208571 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:25:28.246008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.245986 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246025 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246044 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246080 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246147 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246195 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246233 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246261 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.246290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246278 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246285 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.246612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246375 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\") pod \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\" (UID: \"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222\") " Nov 02 22:25:28.246612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246389 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.246612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246426 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.246826 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.246721 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.247201 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247179 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247207 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247223 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247239 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247252 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247270 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247284 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247299 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxckq\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-kube-api-access-mxckq\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.247314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247314 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247328 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247341 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247353 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247366 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.247381 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.248682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.248652 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.248792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.248748 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "kube-api-access-9rr77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.248792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.248757 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.248792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.248770 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token" pod "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" (UID: "1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.348274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348087 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.348274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348232 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.348274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348285 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.349031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348629 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.349031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348872 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.349031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.348917 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.349031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349013 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.349286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349050 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.349286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349080 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.349286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349145 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.349286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349209 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.349286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349237 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349287 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349341 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349396 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349436 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349535 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349597 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\") pod \"8033aec2-77b5-433d-b47f-99679262bc56\" (UID: \"8033aec2-77b5-433d-b47f-99679262bc56\") " Nov 02 22:25:28.350220 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.349675 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\") pod \"7b222d68-7e69-4776-ac81-97ae535841fb\" (UID: \"7b222d68-7e69-4776-ac81-97ae535841fb\") " Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350260 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350559 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rr77\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-kube-api-access-9rr77\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350604 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350621 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350637 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350653 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350666 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.350938 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351083 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351091 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351135 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351221 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351558 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351764 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.353302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.351801 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.354451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.353466 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "kube-api-access-9lv9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.354451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.353672 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.354451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.353817 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.354970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.354872 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.355426 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.355396 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.355523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.355449 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq" pod "7b222d68-7e69-4776-ac81-97ae535841fb" (UID: "7b222d68-7e69-4776-ac81-97ae535841fb"). InnerVolumeSpecName "kube-api-access-qt5nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.356081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.356052 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy" pod "8033aec2-77b5-433d-b47f-99679262bc56" (UID: "8033aec2-77b5-433d-b47f-99679262bc56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.451713 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451689 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451713 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451713 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b222d68-7e69-4776-ac81-97ae535841fb-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451724 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451732 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451741 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451750 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451759 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b222d68-7e69-4776-ac81-97ae535841fb-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451766 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451774 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451783 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451791 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lv9q\" (UniqueName: \"kubernetes.io/projected/8033aec2-77b5-433d-b47f-99679262bc56-kube-api-access-9lv9q\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451799 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8033aec2-77b5-433d-b47f-99679262bc56-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451807 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b222d68-7e69-4776-ac81-97ae535841fb-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451816 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8033aec2-77b5-433d-b47f-99679262bc56-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451824 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8033aec2-77b5-433d-b47f-99679262bc56-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.451870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.451832 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5nq\" (UniqueName: \"kubernetes.io/projected/7b222d68-7e69-4776-ac81-97ae535841fb-kube-api-access-qt5nq\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.523750 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.523718 5859 generic.go:356] "Generic (PLEG): container finished" podID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerID="bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c" exitCode=0 Nov 02 22:25:28.523901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.523768 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" Nov 02 22:25:28.523901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.523799 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" event={"ID":"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23","Type":"ContainerDied","Data":"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c"} Nov 02 22:25:28.523901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.523851 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj" event={"ID":"da5a4e23-2b6f-48e6-970b-1eacc5b5cf23","Type":"ContainerDied","Data":"e34c8a6bad2e391601699f17e66c6429d4b451f92c6ee6f697c3156c0cef5a6c"} Nov 02 22:25:28.523901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.523884 5859 scope.go:122] "RemoveContainer" containerID="bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c" Nov 02 22:25:28.525399 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.525372 5859 generic.go:356] "Generic (PLEG): container finished" podID="7b222d68-7e69-4776-ac81-97ae535841fb" containerID="720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e" exitCode=0 Nov 02 22:25:28.525514 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.525420 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" Nov 02 22:25:28.525514 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.525468 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" event={"ID":"7b222d68-7e69-4776-ac81-97ae535841fb","Type":"ContainerDied","Data":"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e"} Nov 02 22:25:28.525637 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.525519 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8" event={"ID":"7b222d68-7e69-4776-ac81-97ae535841fb","Type":"ContainerDied","Data":"60cd60f1c00d4eac970e20d808d0446a88d88251c9ffe4153326d36a9fc6104f"} Nov 02 22:25:28.527076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.527052 5859 generic.go:356] "Generic (PLEG): container finished" podID="8033aec2-77b5-433d-b47f-99679262bc56" containerID="52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd" exitCode=0 Nov 02 22:25:28.527253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.527089 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" Nov 02 22:25:28.527253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.527130 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" event={"ID":"8033aec2-77b5-433d-b47f-99679262bc56","Type":"ContainerDied","Data":"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd"} Nov 02 22:25:28.527253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.527171 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f" event={"ID":"8033aec2-77b5-433d-b47f-99679262bc56","Type":"ContainerDied","Data":"3af51fe3fb45ac273b7544897d97d402d3d81ebcb760b95dbd2f1e18f8120558"} Nov 02 22:25:28.528823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.528799 5859 generic.go:356] "Generic (PLEG): container finished" podID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerID="870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88" exitCode=0 Nov 02 22:25:28.528971 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.528846 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" Nov 02 22:25:28.528971 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.528858 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" event={"ID":"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222","Type":"ContainerDied","Data":"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88"} Nov 02 22:25:28.528971 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.528896 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp" event={"ID":"1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222","Type":"ContainerDied","Data":"4d4c655feabf9e7aa929b74410f1798568eaaef162206f8ec9c8ab7e2e4ff751"} Nov 02 22:25:28.537350 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.537333 5859 scope.go:122] "RemoveContainer" containerID="bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c" Nov 02 22:25:28.537663 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:28.537634 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c\": container with ID starting with bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c not found: ID does not exist" containerID="bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c" Nov 02 22:25:28.537743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.537670 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c"} err="failed to get container status \"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c\": rpc error: code = NotFound desc = could not find container \"bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c\": container with ID starting with bdd9ddf69ae5577c8ccf33d24fffb2c7de25ad3ed68d6be87ae5640a5f1a750c not found: ID does not exist" Nov 02 22:25:28.537743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.537697 5859 scope.go:122] "RemoveContainer" containerID="720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e" Nov 02 22:25:28.547449 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.547431 5859 scope.go:122] "RemoveContainer" containerID="720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e" Nov 02 22:25:28.547709 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:28.547687 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e\": container with ID starting with 720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e not found: ID does not exist" containerID="720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e" Nov 02 22:25:28.547806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.547714 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e"} err="failed to get container status \"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e\": rpc error: code = NotFound desc = could not find container \"720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e\": container with ID starting with 720badcf60d7b1de4a0f973965025d9d8605a7787200166cf93e211f3979367e not found: ID does not exist" Nov 02 22:25:28.547806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.547728 5859 scope.go:122] "RemoveContainer" containerID="52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd" Nov 02 22:25:28.556719 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.556699 5859 scope.go:122] "RemoveContainer" containerID="52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd" Nov 02 22:25:28.557004 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:28.556987 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd\": container with ID starting with 52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd not found: ID does not exist" containerID="52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd" Nov 02 22:25:28.557071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.557011 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd"} err="failed to get container status \"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd\": rpc error: code = NotFound desc = could not find container \"52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd\": container with ID starting with 52bfed1821f90c0d855a84f99fc8d81f27de3d7d7c8a64b4ea1bc43d91fcf1dd not found: ID does not exist" Nov 02 22:25:28.557071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.557028 5859 scope.go:122] "RemoveContainer" containerID="870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88" Nov 02 22:25:28.572257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.572233 5859 scope.go:122] "RemoveContainer" containerID="870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88" Nov 02 22:25:28.572704 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:28.572675 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88\": container with ID starting with 870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88 not found: ID does not exist" containerID="870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88" Nov 02 22:25:28.572797 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.572714 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88"} err="failed to get container status \"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88\": rpc error: code = NotFound desc = could not find container \"870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88\": container with ID starting with 870287e068453120c253404a7733fadaf518ab6fd3606a39e5ce93e9ca387e88 not found: ID does not exist" Nov 02 22:25:28.665977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.665884 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:25:28.669769 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.669745 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-njcp8"] Nov 02 22:25:28.771334 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.771291 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:25:28.774787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.774760 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fwf6f"] Nov 02 22:25:28.875911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.875883 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:25:28.883910 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.883885 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rpfpj"] Nov 02 22:25:28.984212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.984147 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:25:28.996259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:28.996223 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-z8bpp"] Nov 02 22:25:30.570612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:30.570582 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" path="/var/lib/kubelet/pods/1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222/volumes" Nov 02 22:25:30.571119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:30.571102 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" path="/var/lib/kubelet/pods/7b222d68-7e69-4776-ac81-97ae535841fb/volumes" Nov 02 22:25:30.571553 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:30.571540 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8033aec2-77b5-433d-b47f-99679262bc56" path="/var/lib/kubelet/pods/8033aec2-77b5-433d-b47f-99679262bc56/volumes" Nov 02 22:25:30.572278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:30.572263 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" path="/var/lib/kubelet/pods/da5a4e23-2b6f-48e6-970b-1eacc5b5cf23/volumes" Nov 02 22:25:32.497675 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:32.497642 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:25:32.498118 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:32.497846 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" containerID="cri-o://1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e" gracePeriod=30 Nov 02 22:25:33.635455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:33.635423 5859 patch_prober.go:29] interesting pod/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:33.635822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:33.635476 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:37.766583 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.766517 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:25:37.854275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854249 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854363 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854403 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854442 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854477 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854555 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854576 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:37.854760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854669 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:37.854760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854602 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854713 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:37.854925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854821 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.854925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.854862 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\") pod \"8395503e-dab6-4946-be2f-f8d4698ddb72\" (UID: \"8395503e-dab6-4946-be2f-f8d4698ddb72\") " Nov 02 22:25:37.855094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855071 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:37.855153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855128 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:37.855399 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855373 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.855399 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855397 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.855546 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855411 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8395503e-dab6-4946-be2f-f8d4698ddb72-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.855546 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855423 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.855546 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.855435 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.856783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.856759 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "kube-api-access-pqpbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:37.856876 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.856813 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:37.857041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.857024 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:37.857105 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.857044 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token" pod "8395503e-dab6-4946-be2f-f8d4698ddb72" (UID: "8395503e-dab6-4946-be2f-f8d4698ddb72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:37.956734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.956707 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.956734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.956732 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqpbq\" (UniqueName: \"kubernetes.io/projected/8395503e-dab6-4946-be2f-f8d4698ddb72-kube-api-access-pqpbq\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.956888 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.956742 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.956888 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:37.956750 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8395503e-dab6-4946-be2f-f8d4698ddb72-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.602592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.602550 5859 generic.go:356] "Generic (PLEG): container finished" podID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerID="1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e" exitCode=0 Nov 02 22:25:38.602592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.602583 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerDied","Data":"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e"} Nov 02 22:25:38.602819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.602606 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" Nov 02 22:25:38.602819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.602630 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg" event={"ID":"8395503e-dab6-4946-be2f-f8d4698ddb72","Type":"ContainerDied","Data":"16f75d02180ad26a4d1ca699e919f84ad39899c97d507c9bab33fe01d756f309"} Nov 02 22:25:38.602819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.602656 5859 scope.go:122] "RemoveContainer" containerID="1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e" Nov 02 22:25:38.613278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.613246 5859 scope.go:122] "RemoveContainer" containerID="aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942" Nov 02 22:25:38.624608 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.624425 5859 scope.go:122] "RemoveContainer" containerID="1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e" Nov 02 22:25:38.624710 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:38.624693 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e\": container with ID starting with 1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e not found: ID does not exist" containerID="1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e" Nov 02 22:25:38.624762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.624718 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e"} err="failed to get container status \"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e\": rpc error: code = NotFound desc = could not find container \"1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e\": container with ID starting with 1a9ca1d24f6917e789e8cc1758c6a5d7cc1885b2873a6038464c429cad10a25e not found: ID does not exist" Nov 02 22:25:38.624762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.624736 5859 scope.go:122] "RemoveContainer" containerID="aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942" Nov 02 22:25:38.625006 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:38.624989 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942\": container with ID starting with aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942 not found: ID does not exist" containerID="aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942" Nov 02 22:25:38.625067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.625010 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942"} err="failed to get container status \"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942\": rpc error: code = NotFound desc = could not find container \"aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942\": container with ID starting with aee3ace27690d091038b263a9d7caacf6a12fc21eafcccfd7ea603d09b24c942 not found: ID does not exist" Nov 02 22:25:38.756247 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.756218 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:25:38.802536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:38.802508 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-db69f7-openshift-default-7bffc8bc5f-7hnvg"] Nov 02 22:25:40.571806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:25:40.571777 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" path="/var/lib/kubelet/pods/8395503e-dab6-4946-be2f-f8d4698ddb72/volumes" Nov 02 22:25:55.945768 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:25:55.945727 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:26:25.945319 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:26:25.945279 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:26:55.944471 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:26:55.944432 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:27:02.101144 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.101111 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:27:02.101676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.101418 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" podUID="5513733d-cdf4-4a34-a804-afdf280ba870" containerName="maas-api" containerID="cri-o://d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e" gracePeriod=30 Nov 02 22:27:02.363790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.363713 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:27:02.482678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.482644 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") pod \"5513733d-cdf4-4a34-a804-afdf280ba870\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " Nov 02 22:27:02.482851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.482734 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\") pod \"5513733d-cdf4-4a34-a804-afdf280ba870\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " Nov 02 22:27:02.482851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.482777 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\" (UniqueName: \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\") pod \"5513733d-cdf4-4a34-a804-afdf280ba870\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " Nov 02 22:27:02.483009 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.482927 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\") pod \"5513733d-cdf4-4a34-a804-afdf280ba870\" (UID: \"5513733d-cdf4-4a34-a804-afdf280ba870\") " Nov 02 22:27:02.483510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.483465 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca" pod "5513733d-cdf4-4a34-a804-afdf280ba870" (UID: "5513733d-cdf4-4a34-a804-afdf280ba870"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:02.486016 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.485979 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs" pod "5513733d-cdf4-4a34-a804-afdf280ba870" (UID: "5513733d-cdf4-4a34-a804-afdf280ba870"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:02.486670 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.486643 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls" pod "5513733d-cdf4-4a34-a804-afdf280ba870" (UID: "5513733d-cdf4-4a34-a804-afdf280ba870"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:02.486890 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.486865 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm" pod "5513733d-cdf4-4a34-a804-afdf280ba870" (UID: "5513733d-cdf4-4a34-a804-afdf280ba870"). InnerVolumeSpecName "kube-api-access-kbntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:02.584160 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.584136 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513733d-cdf4-4a34-a804-afdf280ba870-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:02.584160 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.584160 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:02.584302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.584170 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5513733d-cdf4-4a34-a804-afdf280ba870-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:02.584302 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:02.584179 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbntm\" (UniqueName: \"kubernetes.io/projected/5513733d-cdf4-4a34-a804-afdf280ba870-kube-api-access-kbntm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.139029 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.138993 5859 generic.go:356] "Generic (PLEG): container finished" podID="5513733d-cdf4-4a34-a804-afdf280ba870" containerID="d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e" exitCode=0 Nov 02 22:27:03.139470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.139054 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" Nov 02 22:27:03.139470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.139067 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" event={"ID":"5513733d-cdf4-4a34-a804-afdf280ba870","Type":"ContainerDied","Data":"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e"} Nov 02 22:27:03.139470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.139127 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2" event={"ID":"5513733d-cdf4-4a34-a804-afdf280ba870","Type":"ContainerDied","Data":"58b5088aa035d3e1cc3a5e81e1a4070c83e6604beeb1860d2eb83c3a536743bb"} Nov 02 22:27:03.139470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.139149 5859 scope.go:122] "RemoveContainer" containerID="d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e" Nov 02 22:27:03.150252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.150235 5859 scope.go:122] "RemoveContainer" containerID="d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e" Nov 02 22:27:03.150525 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:03.150507 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e\": container with ID starting with d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e not found: ID does not exist" containerID="d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e" Nov 02 22:27:03.150576 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.150531 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e"} err="failed to get container status \"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e\": rpc error: code = NotFound desc = could not find container \"d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e\": container with ID starting with d8a54df5da1bf5abafb20be96d4eaeef92a64d93fcf0284e8f3fb3a012c8760e not found: ID does not exist" Nov 02 22:27:03.293404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.293382 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:27:03.296577 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:03.296556 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3edc8c-5dfff54bf7-9ppb2"] Nov 02 22:27:04.570650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:04.570613 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5513733d-cdf4-4a34-a804-afdf280ba870" path="/var/lib/kubelet/pods/5513733d-cdf4-4a34-a804-afdf280ba870/volumes" Nov 02 22:27:25.944935 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:25.944900 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:27:36.124536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.124460 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 22:27:36.125074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.124697 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" podUID="72bc61a2-4e9d-4b20-959e-23dd4367d740" containerName="maas-api" containerID="cri-o://26fe81364d28e3d0545da9f0d5df0bff56d7a1673dcab32b77e855c836a6bacc" gracePeriod=30 Nov 02 22:27:36.363582 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.363554 5859 generic.go:356] "Generic (PLEG): container finished" podID="72bc61a2-4e9d-4b20-959e-23dd4367d740" containerID="26fe81364d28e3d0545da9f0d5df0bff56d7a1673dcab32b77e855c836a6bacc" exitCode=0 Nov 02 22:27:36.363744 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.363629 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" event={"ID":"72bc61a2-4e9d-4b20-959e-23dd4367d740","Type":"ContainerDied","Data":"26fe81364d28e3d0545da9f0d5df0bff56d7a1673dcab32b77e855c836a6bacc"} Nov 02 22:27:36.390581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.390528 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 22:27:36.438770 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.438747 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\" (UniqueName: \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\") pod \"72bc61a2-4e9d-4b20-959e-23dd4367d740\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " Nov 02 22:27:36.438893 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.438879 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\") pod \"72bc61a2-4e9d-4b20-959e-23dd4367d740\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " Nov 02 22:27:36.439064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.439046 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") pod \"72bc61a2-4e9d-4b20-959e-23dd4367d740\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " Nov 02 22:27:36.439163 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.439085 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\") pod \"72bc61a2-4e9d-4b20-959e-23dd4367d740\" (UID: \"72bc61a2-4e9d-4b20-959e-23dd4367d740\") " Nov 02 22:27:36.439375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.439347 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca" pod "72bc61a2-4e9d-4b20-959e-23dd4367d740" (UID: "72bc61a2-4e9d-4b20-959e-23dd4367d740"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:36.439585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.439563 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bc61a2-4e9d-4b20-959e-23dd4367d740-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:36.441340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.441317 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs" pod "72bc61a2-4e9d-4b20-959e-23dd4367d740" (UID: "72bc61a2-4e9d-4b20-959e-23dd4367d740"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:36.441471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.441456 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls" pod "72bc61a2-4e9d-4b20-959e-23dd4367d740" (UID: "72bc61a2-4e9d-4b20-959e-23dd4367d740"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:36.442106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.442084 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5" pod "72bc61a2-4e9d-4b20-959e-23dd4367d740" (UID: "72bc61a2-4e9d-4b20-959e-23dd4367d740"). InnerVolumeSpecName "kube-api-access-zw7p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:36.540979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.540930 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:36.540979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.540979 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72bc61a2-4e9d-4b20-959e-23dd4367d740-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:36.541249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:36.540994 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw7p5\" (UniqueName: \"kubernetes.io/projected/72bc61a2-4e9d-4b20-959e-23dd4367d740-kube-api-access-zw7p5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:37.371575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:37.371547 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" Nov 02 22:27:37.372026 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:37.371588 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk" event={"ID":"72bc61a2-4e9d-4b20-959e-23dd4367d740","Type":"ContainerDied","Data":"08a845c65227249911a75327e2f1b6664e5d953cde123094bb700353e7289afb"} Nov 02 22:27:37.372026 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:37.371651 5859 scope.go:122] "RemoveContainer" containerID="26fe81364d28e3d0545da9f0d5df0bff56d7a1673dcab32b77e855c836a6bacc" Nov 02 22:27:37.524924 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:37.524896 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 22:27:37.570942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:37.570919 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06e461-b6fd45f7d-ttlgk"] Nov 02 22:27:38.571868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:38.571837 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bc61a2-4e9d-4b20-959e-23dd4367d740" path="/var/lib/kubelet/pods/72bc61a2-4e9d-4b20-959e-23dd4367d740/volumes" Nov 02 22:27:40.884163 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:40.884119 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 22:27:40.884631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:40.884390 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" containerID="cri-o://9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb" gracePeriod=30 Nov 02 22:27:46.156277 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.156219 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 22:27:46.251873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.251847 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252032 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.251905 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252052 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252121 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252091 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252183 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:46.252251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252187 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252283 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252368 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252336 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:46.252368 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252346 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252380 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:46.252471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252405 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252446 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\") pod \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\" (UID: \"884afae4-99a7-4b0a-8fcf-bde5f08004d1\") " Nov 02 22:27:46.252630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252524 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:46.252630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252555 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:46.252928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252904 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.253049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252930 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.253049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252944 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.253049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252974 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.253049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.252987 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.254147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.254123 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:46.254412 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.254392 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:46.254475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.254460 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:46.254595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.254578 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578" pod "884afae4-99a7-4b0a-8fcf-bde5f08004d1" (UID: "884afae4-99a7-4b0a-8fcf-bde5f08004d1"). InnerVolumeSpecName "kube-api-access-pc578". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:46.354640 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.354613 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.354640 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.354638 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.354812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.354649 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884afae4-99a7-4b0a-8fcf-bde5f08004d1-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.354812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.354659 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc578\" (UniqueName: \"kubernetes.io/projected/884afae4-99a7-4b0a-8fcf-bde5f08004d1-kube-api-access-pc578\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:46.438093 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.438028 5859 generic.go:356] "Generic (PLEG): container finished" podID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerID="9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb" exitCode=0 Nov 02 22:27:46.438093 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.438080 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" Nov 02 22:27:46.438282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.438111 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerDied","Data":"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb"} Nov 02 22:27:46.438282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.438255 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx" event={"ID":"884afae4-99a7-4b0a-8fcf-bde5f08004d1","Type":"ContainerDied","Data":"4f195a0054c590e97b751ede86b98511ee4be38939a737d0418f1c4ae211924b"} Nov 02 22:27:46.438384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.438288 5859 scope.go:122] "RemoveContainer" containerID="9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb" Nov 02 22:27:46.451289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.451268 5859 scope.go:122] "RemoveContainer" containerID="c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5" Nov 02 22:27:46.460780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.460764 5859 scope.go:122] "RemoveContainer" containerID="9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb" Nov 02 22:27:46.461076 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:46.461057 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb\": container with ID starting with 9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb not found: ID does not exist" containerID="9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb" Nov 02 22:27:46.461141 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.461087 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb"} err="failed to get container status \"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb\": rpc error: code = NotFound desc = could not find container \"9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb\": container with ID starting with 9ca6373e65a579cf091116186e79834b590c61560b8af1a9b374246e83461feb not found: ID does not exist" Nov 02 22:27:46.461141 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.461114 5859 scope.go:122] "RemoveContainer" containerID="c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5" Nov 02 22:27:46.461339 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:46.461323 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5\": container with ID starting with c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5 not found: ID does not exist" containerID="c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5" Nov 02 22:27:46.461388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.461343 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5"} err="failed to get container status \"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5\": rpc error: code = NotFound desc = could not find container \"c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5\": container with ID starting with c90bf1ce25fcabbd77e61e3b5ef3630a643f7e0ff33c9306f2d6eb38730a1fe5 not found: ID does not exist" Nov 02 22:27:46.594001 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.593977 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 22:27:46.597366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:46.597347 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06e461-openshift-default-d76879f79-k48jx"] Nov 02 22:27:48.571062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:48.571032 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" path="/var/lib/kubelet/pods/884afae4-99a7-4b0a-8fcf-bde5f08004d1/volumes" Nov 02 22:27:49.746033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.745997 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:27:49.746795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746775 5859 cpu_manager.go:414] "containerMap: removing container" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" Nov 02 22:27:49.746795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746795 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746819 5859 cpu_manager.go:414] "containerMap: removing container" podUID="72bc61a2-4e9d-4b20-959e-23dd4367d740" containerName="maas-api" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746827 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bc61a2-4e9d-4b20-959e-23dd4367d740" containerName="maas-api" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746850 5859 cpu_manager.go:414] "containerMap: removing container" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746859 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746869 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746876 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746889 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" containerName="maas-api" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746895 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" containerName="maas-api" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746904 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746911 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746928 5859 cpu_manager.go:414] "containerMap: removing container" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746935 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746964 5859 cpu_manager.go:414] "containerMap: removing container" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746971 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746987 5859 cpu_manager.go:414] "containerMap: removing container" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.747005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.746994 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747025 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5513733d-cdf4-4a34-a804-afdf280ba870" containerName="maas-api" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747032 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5513733d-cdf4-4a34-a804-afdf280ba870" containerName="maas-api" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747043 5859 cpu_manager.go:414] "containerMap: removing container" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747050 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747061 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7689b141-cf35-4fe8-8803-da9250b08bc8" containerName="payload-pre-processing" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747068 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7689b141-cf35-4fe8-8803-da9250b08bc8" containerName="payload-pre-processing" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747208 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5513733d-cdf4-4a34-a804-afdf280ba870" containerName="maas-api" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747221 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747232 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7689b141-cf35-4fe8-8803-da9250b08bc8" containerName="payload-pre-processing" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747244 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8033aec2-77b5-433d-b47f-99679262bc56" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747256 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bc61a2-4e9d-4b20-959e-23dd4367d740" containerName="maas-api" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747267 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3e8f0c-0f0e-49aa-b36e-0789cc3c7222" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747276 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2dbd68-06a6-4370-a58e-888d0e0761c3" containerName="maas-api" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747284 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b222d68-7e69-4776-ac81-97ae535841fb" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747295 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747305 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5a4e23-2b6f-48e6-970b-1eacc5b5cf23" containerName="istio-proxy" Nov 02 22:27:49.747790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.747314 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8395503e-dab6-4946-be2f-f8d4698ddb72" containerName="istio-proxy" Nov 02 22:27:49.752918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.752899 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.805425 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.805390 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:27:49.827970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.827918 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"openshift-service-ca.crt\"" Nov 02 22:27:49.847605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.847578 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"kube-root-ca.crt\"" Nov 02 22:27:49.847605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.847600 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"default-dockercfg-h5psl\"" Nov 02 22:27:49.847801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.847709 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-97995b\"/\"sub-test-model-97995b-kserve-self-signed-certs\"" Nov 02 22:27:49.893689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.893665 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.893824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.893713 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.893873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.893842 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wwn\" (UniqueName: \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.893909 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.893882 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.894039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.894017 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.894114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.894102 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995011 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.994938 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995073 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995139 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995199 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995303 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wwn\" (UniqueName: \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995352 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.995831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.995808 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.997826 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.997762 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:49.998240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:49.998218 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:50.017650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.017624 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:27:50.017902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.017877 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" containerID="cri-o://c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154" gracePeriod=30 Nov 02 22:27:50.018913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.018856 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:50.023474 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.023453 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:50.051997 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.051943 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wwn\" (UniqueName: \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\") pod \"sub-test-model-97995b-kserve-d997984cf-6dpbm\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:50.065591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.065575 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:50.449274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.449240 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:27:50.454000 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:27:50.453975 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9e8bb29_e531_43d1_b10d_cd8360c8d728.slice/crio-2a45fa94627761d8a55b3f2de17a3b31532021b677f886066c36a818ef978b5f WatchSource:0}: Error finding container 2a45fa94627761d8a55b3f2de17a3b31532021b677f886066c36a818ef978b5f: Status 404 returned error can't find the container with id 2a45fa94627761d8a55b3f2de17a3b31532021b677f886066c36a818ef978b5f Nov 02 22:27:50.466695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:50.466668 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" event={"ID":"c9e8bb29-e531-43d1-b10d-cd8360c8d728","Type":"ContainerStarted","Data":"2a45fa94627761d8a55b3f2de17a3b31532021b677f886066c36a818ef978b5f"} Nov 02 22:27:51.474572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:51.474539 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" event={"ID":"c9e8bb29-e531-43d1-b10d-cd8360c8d728","Type":"ContainerStarted","Data":"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c"} Nov 02 22:27:51.583031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:51.582980 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" podStartSLOduration=2.5829672280000002 podStartE2EDuration="2.582967228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:51.580503346 +0000 UTC m=+4711.128753142" watchObservedRunningTime="2026-11-02 22:27:51.582967228 +0000 UTC m=+4711.131217006" Nov 02 22:27:52.065727 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:52.065693 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:27:55.284151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.284091 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:27:55.356394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356361 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356401 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356434 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356480 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356527 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356617 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356661 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356674 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:55.356856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356701 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.356856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356698 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:55.356856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356801 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:55.356856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356811 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\") pod \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\" (UID: \"416c9c5e-19f0-4a38-976b-c5c74c90ed66\") " Nov 02 22:27:55.357103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.356890 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:55.357103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357085 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:55.357377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357351 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.357501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357379 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.357501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357393 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.357501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357408 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.357501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.357421 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.358701 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.358676 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "kube-api-access-74x7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:55.358827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.358782 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:55.358944 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.358925 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:55.359091 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.359013 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token" pod "416c9c5e-19f0-4a38-976b-c5c74c90ed66" (UID: "416c9c5e-19f0-4a38-976b-c5c74c90ed66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:55.459132 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.459099 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.459132 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.459134 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x7w\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-kube-api-access-74x7w\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.459339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.459149 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.459339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.459164 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416c9c5e-19f0-4a38-976b-c5c74c90ed66-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:55.502088 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.502052 5859 generic.go:356] "Generic (PLEG): container finished" podID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerID="c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154" exitCode=0 Nov 02 22:27:55.502276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.502111 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" Nov 02 22:27:55.502276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.502138 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerDied","Data":"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154"} Nov 02 22:27:55.502276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.502201 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc" event={"ID":"416c9c5e-19f0-4a38-976b-c5c74c90ed66","Type":"ContainerDied","Data":"4f7638946d80b506af0d6e911adc2ebf9266e4cba93cfbc0ded38e0ddc8fad06"} Nov 02 22:27:55.502276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.502229 5859 scope.go:122] "RemoveContainer" containerID="c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154" Nov 02 22:27:55.513698 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.513678 5859 scope.go:122] "RemoveContainer" containerID="59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3" Nov 02 22:27:55.523333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.523294 5859 scope.go:122] "RemoveContainer" containerID="c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154" Nov 02 22:27:55.523611 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:55.523585 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154\": container with ID starting with c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154 not found: ID does not exist" containerID="c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154" Nov 02 22:27:55.523708 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.523617 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154"} err="failed to get container status \"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154\": rpc error: code = NotFound desc = could not find container \"c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154\": container with ID starting with c0d55e67794f8e3a8085d28f8bc4950ff7000a47380e87beb15edd6a0ed8f154 not found: ID does not exist" Nov 02 22:27:55.523708 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.523634 5859 scope.go:122] "RemoveContainer" containerID="59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3" Nov 02 22:27:55.523932 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:55.523911 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3\": container with ID starting with 59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3 not found: ID does not exist" containerID="59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3" Nov 02 22:27:55.524038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.523942 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3"} err="failed to get container status \"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3\": rpc error: code = NotFound desc = could not find container \"59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3\": container with ID starting with 59d593cb6555e3de14ea2cbecb506512d1017784c4acbbf5803c3a2a097f5fc3 not found: ID does not exist" Nov 02 22:27:55.656053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.656021 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:27:55.660653 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:55.660566 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dd28e6-openshift-default-8d476cb9-4prjc"] Nov 02 22:27:55.948305 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:27:55.948220 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:27:56.571495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:56.571462 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" path="/var/lib/kubelet/pods/416c9c5e-19f0-4a38-976b-c5c74c90ed66/volumes" Nov 02 22:27:58.901218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:27:58.901187 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:28:00.066290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:00.066259 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:28:00.078865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:00.078840 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:28:00.546008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:00.545978 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:28:03.902500 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:03.902468 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" podUID="f9c0cf1d-ed63-4049-be16-b959ff929931" containerName="payload-processing" containerID="cri-o://33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91" gracePeriod=25 Nov 02 22:28:04.160939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.160886 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:28:04.256878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.256855 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\") pod \"f9c0cf1d-ed63-4049-be16-b959ff929931\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " Nov 02 22:28:04.257042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.256976 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\" (UniqueName: \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\") pod \"f9c0cf1d-ed63-4049-be16-b959ff929931\" (UID: \"f9c0cf1d-ed63-4049-be16-b959ff929931\") " Nov 02 22:28:04.257226 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.257204 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume" pod "f9c0cf1d-ed63-4049-be16-b959ff929931" (UID: "f9c0cf1d-ed63-4049-be16-b959ff929931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:04.258994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.258941 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h" pod "f9c0cf1d-ed63-4049-be16-b959ff929931" (UID: "f9c0cf1d-ed63-4049-be16-b959ff929931"). InnerVolumeSpecName "kube-api-access-54k5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:04.358686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.358660 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c0cf1d-ed63-4049-be16-b959ff929931-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:04.358686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.358685 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k5h\" (UniqueName: \"kubernetes.io/projected/f9c0cf1d-ed63-4049-be16-b959ff929931-kube-api-access-54k5h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:04.562396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.562333 5859 generic.go:356] "Generic (PLEG): container finished" podID="f9c0cf1d-ed63-4049-be16-b959ff929931" containerID="33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91" exitCode=0 Nov 02 22:28:04.562396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.562382 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" Nov 02 22:28:04.562571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.562397 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" event={"ID":"f9c0cf1d-ed63-4049-be16-b959ff929931","Type":"ContainerDied","Data":"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91"} Nov 02 22:28:04.562571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.562460 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr" event={"ID":"f9c0cf1d-ed63-4049-be16-b959ff929931","Type":"ContainerDied","Data":"82811ca9668a3be41662038b975b6b0e48b3336c274aed00837f49dba315b7fc"} Nov 02 22:28:04.562571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.562481 5859 scope.go:122] "RemoveContainer" containerID="33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91" Nov 02 22:28:04.573259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.573237 5859 scope.go:122] "RemoveContainer" containerID="33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91" Nov 02 22:28:04.573678 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:04.573660 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91\": container with ID starting with 33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91 not found: ID does not exist" containerID="33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91" Nov 02 22:28:04.573753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.573688 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91"} err="failed to get container status \"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91\": rpc error: code = NotFound desc = could not find container \"33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91\": container with ID starting with 33bde4c1b2aa2e349647db6ff2a4ae4911b7e050a394b25b9585555a24e08d91 not found: ID does not exist" Nov 02 22:28:04.715044 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.715015 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:28:04.719129 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:04.719108 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf2a53-6565465b4c-sbpwr"] Nov 02 22:28:06.570475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:06.570444 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c0cf1d-ed63-4049-be16-b959ff929931" path="/var/lib/kubelet/pods/f9c0cf1d-ed63-4049-be16-b959ff929931/volumes" Nov 02 22:28:08.656721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:08.656690 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:28:23.678041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:23.677989 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" podUID="c5924f8f-bece-4802-b555-b7164ca81fd6" containerName="main" containerID="cri-o://75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241" gracePeriod=45 Nov 02 22:28:23.730990 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:23.730961 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5924f8f_bece_4802_b555_b7164ca81fd6.slice/crio-conmon-75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5924f8f_bece_4802_b555_b7164ca81fd6.slice/crio-75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:28:23.938216 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:23.938159 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:28:24.057485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057456 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.057631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057552 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.057631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057573 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.057763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057648 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.057763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057685 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:24.057763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057699 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\" (UniqueName: \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.057922 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057818 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\") pod \"c5924f8f-bece-4802-b555-b7164ca81fd6\" (UID: \"c5924f8f-bece-4802-b555-b7164ca81fd6\") " Nov 02 22:28:24.058028 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.057996 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:24.058109 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.058083 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:24.058372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.058354 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.058445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.058378 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.058445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.058391 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.059914 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.059888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:24.060171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.060151 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:24.060257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.060191 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt" pod "c5924f8f-bece-4802-b555-b7164ca81fd6" (UID: "c5924f8f-bece-4802-b555-b7164ca81fd6"). InnerVolumeSpecName "kube-api-access-985kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:24.159489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.159453 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5924f8f-bece-4802-b555-b7164ca81fd6-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.159489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.159490 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5924f8f-bece-4802-b555-b7164ca81fd6-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.159489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.159505 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-985kt\" (UniqueName: \"kubernetes.io/projected/c5924f8f-bece-4802-b555-b7164ca81fd6-kube-api-access-985kt\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:24.680481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.680450 5859 generic.go:356] "Generic (PLEG): container finished" podID="c5924f8f-bece-4802-b555-b7164ca81fd6" containerID="75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241" exitCode=0 Nov 02 22:28:24.680851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.680493 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" Nov 02 22:28:24.680851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.680532 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" event={"ID":"c5924f8f-bece-4802-b555-b7164ca81fd6","Type":"ContainerDied","Data":"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241"} Nov 02 22:28:24.680851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.680576 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2" event={"ID":"c5924f8f-bece-4802-b555-b7164ca81fd6","Type":"ContainerDied","Data":"801286d1d3184175aac7789c105fbefe5374fb177f978c01373ccc2e770cb883"} Nov 02 22:28:24.680851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.680598 5859 scope.go:122] "RemoveContainer" containerID="75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241" Nov 02 22:28:24.689926 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.689909 5859 scope.go:122] "RemoveContainer" containerID="75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241" Nov 02 22:28:24.690232 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:24.690212 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241\": container with ID starting with 75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241 not found: ID does not exist" containerID="75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241" Nov 02 22:28:24.690285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.690239 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241"} err="failed to get container status \"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241\": rpc error: code = NotFound desc = could not find container \"75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241\": container with ID starting with 75f287f10f667e890974df776963aea6fd68653db3324c2a2b5efa3b1af2b241 not found: ID does not exist" Nov 02 22:28:24.835290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.835208 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:28:24.880298 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.880273 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7cde84/sub-test-model-7cde84-kserve-6678458df5-p9rz2"] Nov 02 22:28:24.980248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:24.980218 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:28:25.944897 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:25.944865 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:28:26.571001 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:26.570969 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5924f8f-bece-4802-b555-b7164ca81fd6" path="/var/lib/kubelet/pods/c5924f8f-bece-4802-b555-b7164ca81fd6/volumes" Nov 02 22:28:31.886170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:31.886139 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 22:28:36.887062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:36.887023 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" containerID="cri-o://452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e" gracePeriod=25 Nov 02 22:28:40.002487 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.002454 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" podUID="22092d4e-5eec-42e8-89e5-99175541f327" containerName="main" containerID="cri-o://50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f" gracePeriod=45 Nov 02 22:28:40.275408 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.275384 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:28:40.414113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414082 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\" (UniqueName: \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414271 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414130 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\" (UniqueName: \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414271 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414237 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414360 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414450 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414434 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414501 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\") pod \"22092d4e-5eec-42e8-89e5-99175541f327\" (UID: \"22092d4e-5eec-42e8-89e5-99175541f327\") " Nov 02 22:28:40.414569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414519 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:40.414659 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414621 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:40.414797 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.414776 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:40.415248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.415220 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.415379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.415251 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.415379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.415267 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.416405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.416381 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:40.416595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.416569 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:40.416703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.416657 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5" pod "22092d4e-5eec-42e8-89e5-99175541f327" (UID: "22092d4e-5eec-42e8-89e5-99175541f327"). InnerVolumeSpecName "kube-api-access-gwtf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:40.516336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.516287 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtf5\" (UniqueName: \"kubernetes.io/projected/22092d4e-5eec-42e8-89e5-99175541f327-kube-api-access-gwtf5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.516336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.516307 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22092d4e-5eec-42e8-89e5-99175541f327-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.516336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.516317 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22092d4e-5eec-42e8-89e5-99175541f327-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:40.780214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.780139 5859 generic.go:356] "Generic (PLEG): container finished" podID="22092d4e-5eec-42e8-89e5-99175541f327" containerID="50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f" exitCode=0 Nov 02 22:28:40.780214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.780187 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" Nov 02 22:28:40.780380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.780217 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" event={"ID":"22092d4e-5eec-42e8-89e5-99175541f327","Type":"ContainerDied","Data":"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f"} Nov 02 22:28:40.780380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.780263 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp" event={"ID":"22092d4e-5eec-42e8-89e5-99175541f327","Type":"ContainerDied","Data":"4702532be801661eaf61d2b3795717195d9cb1f79d4c4296bf8337df07f0c50e"} Nov 02 22:28:40.780380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.780280 5859 scope.go:122] "RemoveContainer" containerID="50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f" Nov 02 22:28:40.790763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.790748 5859 scope.go:122] "RemoveContainer" containerID="50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f" Nov 02 22:28:40.791052 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:40.791032 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f\": container with ID starting with 50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f not found: ID does not exist" containerID="50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f" Nov 02 22:28:40.791129 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.791059 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f"} err="failed to get container status \"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f\": rpc error: code = NotFound desc = could not find container \"50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f\": container with ID starting with 50db6ffc74d9a526627fa6a76e074cbf6ca35942be53a8d9c96e58787a5a498f not found: ID does not exist" Nov 02 22:28:40.934003 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.933975 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:28:40.937084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:40.937065 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-58ca7f/sub-test-model-58ca7f-kserve-b6794d97c-2ssdp"] Nov 02 22:28:42.570063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:42.570026 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22092d4e-5eec-42e8-89e5-99175541f327" path="/var/lib/kubelet/pods/22092d4e-5eec-42e8-89e5-99175541f327/volumes" Nov 02 22:28:45.230040 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:45.230005 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:55.229942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:55.229903 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:28:55.943786 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:55.943752 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:28:56.633775 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:56.633739 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 22:28:56.634276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:56.633998 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" podUID="ac857156-a817-4303-a981-5c741136a896" containerName="maas-api" containerID="cri-o://9fe16f2e1e71d987ab596e9eff818a5e42f835fa361f6056772bfaf507c43b51" gracePeriod=30 Nov 02 22:28:56.882692 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:56.882664 5859 generic.go:356] "Generic (PLEG): container finished" podID="ac857156-a817-4303-a981-5c741136a896" containerID="9fe16f2e1e71d987ab596e9eff818a5e42f835fa361f6056772bfaf507c43b51" exitCode=0 Nov 02 22:28:56.882839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:56.882738 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" event={"ID":"ac857156-a817-4303-a981-5c741136a896","Type":"ContainerDied","Data":"9fe16f2e1e71d987ab596e9eff818a5e42f835fa361f6056772bfaf507c43b51"} Nov 02 22:28:56.991536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:56.991512 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 22:28:57.156985 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.156961 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 22:28:57.181115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181089 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\") pod \"ac857156-a817-4303-a981-5c741136a896\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " Nov 02 22:28:57.181230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181185 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") pod \"ac857156-a817-4303-a981-5c741136a896\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " Nov 02 22:28:57.181280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181226 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\" (UniqueName: \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\") pod \"ac857156-a817-4303-a981-5c741136a896\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " Nov 02 22:28:57.181349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181329 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\") pod \"ac857156-a817-4303-a981-5c741136a896\" (UID: \"ac857156-a817-4303-a981-5c741136a896\") " Nov 02 22:28:57.181674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181646 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca" pod "ac857156-a817-4303-a981-5c741136a896" (UID: "ac857156-a817-4303-a981-5c741136a896"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:57.181825 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.181804 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac857156-a817-4303-a981-5c741136a896-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.183076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.183053 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs" pod "ac857156-a817-4303-a981-5c741136a896" (UID: "ac857156-a817-4303-a981-5c741136a896"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:57.183230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.183214 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5" pod "ac857156-a817-4303-a981-5c741136a896" (UID: "ac857156-a817-4303-a981-5c741136a896"). InnerVolumeSpecName "kube-api-access-bvkf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:57.183294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.183278 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls" pod "ac857156-a817-4303-a981-5c741136a896" (UID: "ac857156-a817-4303-a981-5c741136a896"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:57.282690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.282663 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\" (UniqueName: \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\") pod \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " Nov 02 22:28:57.282858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.282819 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\") pod \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\" (UID: \"58fb04b2-2384-4c92-ac02-15a6c5c67eab\") " Nov 02 22:28:57.283148 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.283128 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.283148 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.283145 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac857156-a817-4303-a981-5c741136a896-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.283332 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.283155 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkf5\" (UniqueName: \"kubernetes.io/projected/ac857156-a817-4303-a981-5c741136a896-kube-api-access-bvkf5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.283332 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.283235 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume" pod "58fb04b2-2384-4c92-ac02-15a6c5c67eab" (UID: "58fb04b2-2384-4c92-ac02-15a6c5c67eab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:57.284712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.284692 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b" pod "58fb04b2-2384-4c92-ac02-15a6c5c67eab" (UID: "58fb04b2-2384-4c92-ac02-15a6c5c67eab"). InnerVolumeSpecName "kube-api-access-k649b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:57.384031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.384001 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb04b2-2384-4c92-ac02-15a6c5c67eab-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.384031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.384023 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k649b\" (UniqueName: \"kubernetes.io/projected/58fb04b2-2384-4c92-ac02-15a6c5c67eab-kube-api-access-k649b\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:57.890289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.890257 5859 generic.go:356] "Generic (PLEG): container finished" podID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerID="452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e" exitCode=0 Nov 02 22:28:57.890743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.890298 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" Nov 02 22:28:57.890743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.890296 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" event={"ID":"58fb04b2-2384-4c92-ac02-15a6c5c67eab","Type":"ContainerDied","Data":"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e"} Nov 02 22:28:57.890743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.890359 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q" event={"ID":"58fb04b2-2384-4c92-ac02-15a6c5c67eab","Type":"ContainerDied","Data":"ac0d196da2203ae17dd883c4e72db57e97ce36f7b14f0eb7fb61a2767f34bd7b"} Nov 02 22:28:57.890743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.890382 5859 scope.go:122] "RemoveContainer" containerID="452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e" Nov 02 22:28:57.891944 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.891924 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" Nov 02 22:28:57.892083 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.891966 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" event={"ID":"ac857156-a817-4303-a981-5c741136a896","Type":"ContainerDied","Data":"dcc015a21ce4ccaaa3fd62b290d4ec11831c68457bf02e91f7afe2eb851455f9"} Nov 02 22:28:57.900722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.900702 5859 scope.go:122] "RemoveContainer" containerID="452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e" Nov 02 22:28:57.901002 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:28:57.900977 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e\": container with ID starting with 452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e not found: ID does not exist" containerID="452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e" Nov 02 22:28:57.901102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.901016 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e"} err="failed to get container status \"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e\": rpc error: code = NotFound desc = could not find container \"452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e\": container with ID starting with 452e8113213321d6722927cd4c090daffe74961f4f3a6589a7829b1150750d4e not found: ID does not exist" Nov 02 22:28:57.901102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:57.901038 5859 scope.go:122] "RemoveContainer" containerID="9fe16f2e1e71d987ab596e9eff818a5e42f835fa361f6056772bfaf507c43b51" Nov 02 22:28:58.052803 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.052760 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 22:28:58.057194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.057168 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-7cde84-c6d9f94f9-t5p7q"] Nov 02 22:28:58.161010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.160929 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 22:28:58.167084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.167054 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp"] Nov 02 22:28:58.570669 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.570638 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" path="/var/lib/kubelet/pods/58fb04b2-2384-4c92-ac02-15a6c5c67eab/volumes" Nov 02 22:28:58.571090 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:58.571076 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac857156-a817-4303-a981-5c741136a896" path="/var/lib/kubelet/pods/ac857156-a817-4303-a981-5c741136a896/volumes" Nov 02 22:28:59.929669 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:28:59.929634 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c7e71f-6f485b6dc6-kmthp" podUID="ac857156-a817-4303-a981-5c741136a896" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.145:8443/health\": context deadline exceeded" Nov 02 22:29:02.422436 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:02.422350 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:29:02.422853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:02.422579 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" containerID="cri-o://d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f" gracePeriod=30 Nov 02 22:29:06.483496 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:06.483455 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 22:29:07.691572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.691521 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:29:07.788146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788122 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788169 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788201 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788221 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788247 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788499 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788285 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788499 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788361 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788492 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:07.788609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788510 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:07.788609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788557 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788635 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\") pod \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\" (UID: \"a30ab5d3-cc73-4fa0-957a-67da0b498a2b\") " Nov 02 22:29:07.788868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788820 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:07.788868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.788830 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:07.789050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789029 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:07.789214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789197 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.789288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789215 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.789288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789225 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.789288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789235 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.789288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.789244 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.790481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.790463 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:07.790571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.790524 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:29:07.790699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.790674 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:07.790795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.790716 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9" pod "a30ab5d3-cc73-4fa0-957a-67da0b498a2b" (UID: "a30ab5d3-cc73-4fa0-957a-67da0b498a2b"). InnerVolumeSpecName "kube-api-access-p6jc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:07.890099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.890071 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jc9\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-kube-api-access-p6jc9\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.890215 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.890104 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.890215 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.890119 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.890215 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.890132 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30ab5d3-cc73-4fa0-957a-67da0b498a2b-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:07.898417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.898390 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 22:29:07.900989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.900938 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 22:29:07.959874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.959819 5859 generic.go:356] "Generic (PLEG): container finished" podID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerID="d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f" exitCode=0 Nov 02 22:29:07.959988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.959871 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" Nov 02 22:29:07.959988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.959879 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerDied","Data":"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f"} Nov 02 22:29:07.960081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.959994 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf" event={"ID":"a30ab5d3-cc73-4fa0-957a-67da0b498a2b","Type":"ContainerDied","Data":"a3fd951f2a7a1fb43f13608671dbdb8d9873e80670eb353ba8075d8baf713b39"} Nov 02 22:29:07.960081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.960026 5859 scope.go:122] "RemoveContainer" containerID="d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f" Nov 02 22:29:07.970062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.970042 5859 scope.go:122] "RemoveContainer" containerID="c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd" Nov 02 22:29:07.981585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.981554 5859 scope.go:122] "RemoveContainer" containerID="d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f" Nov 02 22:29:07.981905 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:07.981888 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f\": container with ID starting with d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f not found: ID does not exist" containerID="d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f" Nov 02 22:29:07.982002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.981911 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f"} err="failed to get container status \"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f\": rpc error: code = NotFound desc = could not find container \"d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f\": container with ID starting with d7c6ce142a006da167083351e4310e7c82f428a503d82797bb91db3259972e8f not found: ID does not exist" Nov 02 22:29:07.982002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.981930 5859 scope.go:122] "RemoveContainer" containerID="c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd" Nov 02 22:29:07.982188 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:07.982171 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd\": container with ID starting with c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd not found: ID does not exist" containerID="c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd" Nov 02 22:29:07.982242 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:07.982192 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd"} err="failed to get container status \"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd\": rpc error: code = NotFound desc = could not find container \"c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd\": container with ID starting with c2dc37d37a4b73a3b61fb614ee0b0013dfe894ee21e3a088ec0b2286cd3394cd not found: ID does not exist" Nov 02 22:29:08.218770 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:08.218710 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:29:08.222509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:08.222481 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cf2a53-openshift-default-6776c6556f-rd6kf"] Nov 02 22:29:08.570897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:08.570870 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" path="/var/lib/kubelet/pods/a30ab5d3-cc73-4fa0-957a-67da0b498a2b/volumes" Nov 02 22:29:21.329652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.329620 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:29:21.330311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330291 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f9c0cf1d-ed63-4049-be16-b959ff929931" containerName="payload-processing" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330313 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c0cf1d-ed63-4049-be16-b959ff929931" containerName="payload-processing" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330339 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ac857156-a817-4303-a981-5c741136a896" containerName="maas-api" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330346 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac857156-a817-4303-a981-5c741136a896" containerName="maas-api" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330357 5859 cpu_manager.go:414] "containerMap: removing container" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330367 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330396 5859 cpu_manager.go:414] "containerMap: removing container" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330402 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330408 5859 cpu_manager.go:414] "containerMap: removing container" podUID="22092d4e-5eec-42e8-89e5-99175541f327" containerName="main" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330412 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22092d4e-5eec-42e8-89e5-99175541f327" containerName="main" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330420 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330424 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.330434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330441 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c5924f8f-bece-4802-b555-b7164ca81fd6" containerName="main" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330445 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5924f8f-bece-4802-b555-b7164ca81fd6" containerName="main" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330451 5859 cpu_manager.go:414] "containerMap: removing container" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330455 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330467 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330471 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330562 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c0cf1d-ed63-4049-be16-b959ff929931" containerName="payload-processing" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330575 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac857156-a817-4303-a981-5c741136a896" containerName="maas-api" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330582 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330595 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="22092d4e-5eec-42e8-89e5-99175541f327" containerName="main" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330602 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330614 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5924f8f-bece-4802-b555-b7164ca81fd6" containerName="main" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330623 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fb04b2-2384-4c92-ac02-15a6c5c67eab" containerName="payload-processing" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330633 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="884afae4-99a7-4b0a-8fcf-bde5f08004d1" containerName="istio-proxy" Nov 02 22:29:21.331043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.330642 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30ab5d3-cc73-4fa0-957a-67da0b498a2b" containerName="istio-proxy" Nov 02 22:29:21.335796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.335774 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.386540 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.386514 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:29:21.426198 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426172 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.426328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426206 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.426328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426226 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.426427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.426475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426430 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7wj\" (UniqueName: \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.426475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.426471 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.430068 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.430048 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"openshift-service-ca.crt\"" Nov 02 22:29:21.430068 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.430050 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"sub-test-model-a1d751-kserve-self-signed-certs\"" Nov 02 22:29:21.431205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.431190 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"default-dockercfg-vmzss\"" Nov 02 22:29:21.431284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.431252 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"kube-root-ca.crt\"" Nov 02 22:29:21.503867 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.503831 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" containerID="cri-o://3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c" gracePeriod=45 Nov 02 22:29:21.527860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.527838 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7wj\" (UniqueName: \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.527983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.527898 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528064 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528105 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528131 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528229 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528460 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528552 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528486 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.528593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.528548 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.530218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.530201 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.530278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.530260 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.581989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.581926 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7wj\" (UniqueName: \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\") pod \"sub-test-model-a1d751-kserve-bfbc97467-pjrtg\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.650340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.650311 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:21.766309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.766288 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 22:29:21.933588 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933514 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.933734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933635 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\" (UniqueName: \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.933777 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933741 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.933777 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933755 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:21.933848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933833 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.933897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933867 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.933939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.933918 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\") pod \"66ed1713-c414-44f6-9370-0fc4a301881a\" (UID: \"66ed1713-c414-44f6-9370-0fc4a301881a\") " Nov 02 22:29:21.934037 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.934006 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:21.934338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.934311 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:21.934462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.934438 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:21.934605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.934465 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:21.934605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.934482 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:21.936051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.936025 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "kube-api-access-lsr7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:21.936145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.936078 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:21.936145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:21.936095 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs" pod "66ed1713-c414-44f6-9370-0fc4a301881a" (UID: "66ed1713-c414-44f6-9370-0fc4a301881a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:22.032549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.032505 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:29:22.035715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.035686 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsr7g\" (UniqueName: \"kubernetes.io/projected/66ed1713-c414-44f6-9370-0fc4a301881a-kube-api-access-lsr7g\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.035861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.035718 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed1713-c414-44f6-9370-0fc4a301881a-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.035861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.035734 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed1713-c414-44f6-9370-0fc4a301881a-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.037026 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:29:22.037001 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fe24530_0cdd_40dd_bef8_774d0c87e8bf.slice/crio-ca58a4ed510fffaf6f608ed03bfdf705a2706e9443f8915697cdbaacafc7e8ad WatchSource:0}: Error finding container ca58a4ed510fffaf6f608ed03bfdf705a2706e9443f8915697cdbaacafc7e8ad: Status 404 returned error can't find the container with id ca58a4ed510fffaf6f608ed03bfdf705a2706e9443f8915697cdbaacafc7e8ad Nov 02 22:29:22.049561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.049528 5859 generic.go:356] "Generic (PLEG): container finished" podID="66ed1713-c414-44f6-9370-0fc4a301881a" containerID="3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c" exitCode=0 Nov 02 22:29:22.049672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.049573 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" Nov 02 22:29:22.049672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.049622 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" event={"ID":"66ed1713-c414-44f6-9370-0fc4a301881a","Type":"ContainerDied","Data":"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c"} Nov 02 22:29:22.049762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.049677 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc" event={"ID":"66ed1713-c414-44f6-9370-0fc4a301881a","Type":"ContainerDied","Data":"94fa6431494fba9dc5fb5a2f10be85397c2f7679366e58653431d9449c2c6702"} Nov 02 22:29:22.049762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.049701 5859 scope.go:122] "RemoveContainer" containerID="3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c" Nov 02 22:29:22.051244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.051220 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" event={"ID":"8fe24530-0cdd-40dd-bef8-774d0c87e8bf","Type":"ContainerStarted","Data":"ca58a4ed510fffaf6f608ed03bfdf705a2706e9443f8915697cdbaacafc7e8ad"} Nov 02 22:29:22.059983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.059920 5859 scope.go:122] "RemoveContainer" containerID="3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c" Nov 02 22:29:22.060556 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:22.060533 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c\": container with ID starting with 3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c not found: ID does not exist" containerID="3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c" Nov 02 22:29:22.060652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.060562 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c"} err="failed to get container status \"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c\": rpc error: code = NotFound desc = could not find container \"3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c\": container with ID starting with 3ea5a47a397ea232a3da5af41414885946e99cc41ad4283b84b7623e98eb960c not found: ID does not exist" Nov 02 22:29:22.204303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.204219 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 22:29:22.208475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.208443 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-w2-6fa533-kserve-8477bf9fc9-lv5tc"] Nov 02 22:29:22.570344 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.570314 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" path="/var/lib/kubelet/pods/66ed1713-c414-44f6-9370-0fc4a301881a/volumes" Nov 02 22:29:22.918303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.918265 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" containerID="cri-o://e3f201ed489bed5cf1087ab95d945184f9554ba18b79fda3345a37540af16431" gracePeriod=45 Nov 02 22:29:22.926391 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.926353 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" containerID="cri-o://66baee28c2984a09d5ac53c6cacfadba9abd1c5f2b2719cabd31d57f2cff92f0" gracePeriod=45 Nov 02 22:29:22.994218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:22.994190 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Nov 02 22:29:23.061896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.061865 5859 generic.go:356] "Generic (PLEG): container finished" podID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerID="66baee28c2984a09d5ac53c6cacfadba9abd1c5f2b2719cabd31d57f2cff92f0" exitCode=0 Nov 02 22:29:23.062100 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.061944 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" event={"ID":"57f246ab-a5d2-46b0-a8fa-8ff822401222","Type":"ContainerDied","Data":"66baee28c2984a09d5ac53c6cacfadba9abd1c5f2b2719cabd31d57f2cff92f0"} Nov 02 22:29:23.068968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.063869 5859 generic.go:356] "Generic (PLEG): container finished" podID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerID="e3f201ed489bed5cf1087ab95d945184f9554ba18b79fda3345a37540af16431" exitCode=0 Nov 02 22:29:23.068968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.064053 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" event={"ID":"9a4997b2-60cb-42ee-b216-e4a93622b784","Type":"ContainerDied","Data":"e3f201ed489bed5cf1087ab95d945184f9554ba18b79fda3345a37540af16431"} Nov 02 22:29:23.068968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.066037 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" event={"ID":"8fe24530-0cdd-40dd-bef8-774d0c87e8bf","Type":"ContainerStarted","Data":"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67"} Nov 02 22:29:23.176172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.176073 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" podStartSLOduration=2.176059441 podStartE2EDuration="2.176059441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:23.175228114 +0000 UTC m=+4802.723477895" watchObservedRunningTime="2026-11-02 22:29:23.176059441 +0000 UTC m=+4802.724309221" Nov 02 22:29:23.190702 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.190678 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 22:29:23.197848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.197827 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 22:29:23.351018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.350984 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351054 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\" (UniqueName: \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351087 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351135 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351182 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351229 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351323 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\" (UniqueName: \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351353 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\" (UniqueName: \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351388 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351441 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351504 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351505 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.351578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351513 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\") pod \"57f246ab-a5d2-46b0-a8fa-8ff822401222\" (UID: \"57f246ab-a5d2-46b0-a8fa-8ff822401222\") " Nov 02 22:29:23.351969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351612 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\") pod \"9a4997b2-60cb-42ee-b216-e4a93622b784\" (UID: \"9a4997b2-60cb-42ee-b216-e4a93622b784\") " Nov 02 22:29:23.351969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351695 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.351969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351727 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.352160 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.351924 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.352224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352195 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.352224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352217 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.352335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352234 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.352335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352249 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.352335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352263 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.352511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.352488 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.354047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.354012 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.354047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.354036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "kube-api-access-pdxzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:23.354290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.354262 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs" pod "57f246ab-a5d2-46b0-a8fa-8ff822401222" (UID: "57f246ab-a5d2-46b0-a8fa-8ff822401222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:23.355020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.355003 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.355358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.355334 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "kube-api-access-rw8vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:23.355489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.355466 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs" pod "9a4997b2-60cb-42ee-b216-e4a93622b784" (UID: "9a4997b2-60cb-42ee-b216-e4a93622b784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453871 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a4997b2-60cb-42ee-b216-e4a93622b784-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453901 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw8vk\" (UniqueName: \"kubernetes.io/projected/9a4997b2-60cb-42ee-b216-e4a93622b784-kube-api-access-rw8vk\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453913 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a4997b2-60cb-42ee-b216-e4a93622b784-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453921 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453929 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f246ab-a5d2-46b0-a8fa-8ff822401222-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453937 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxzf\" (UniqueName: \"kubernetes.io/projected/57f246ab-a5d2-46b0-a8fa-8ff822401222-kube-api-access-pdxzf\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.453964 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.453945 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f246ab-a5d2-46b0-a8fa-8ff822401222-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.651274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:23.651247 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:24.073564 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.073539 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" Nov 02 22:29:24.073734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.073538 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl" event={"ID":"57f246ab-a5d2-46b0-a8fa-8ff822401222","Type":"ContainerDied","Data":"6a1e166affebaaabdba30dfd8e0b884627cf1c97cf1c8f2437ca574d1b4dbf2d"} Nov 02 22:29:24.073734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.073658 5859 scope.go:122] "RemoveContainer" containerID="66baee28c2984a09d5ac53c6cacfadba9abd1c5f2b2719cabd31d57f2cff92f0" Nov 02 22:29:24.075076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.075060 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" Nov 02 22:29:24.075175 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.075130 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd" event={"ID":"9a4997b2-60cb-42ee-b216-e4a93622b784","Type":"ContainerDied","Data":"1461f134e37d30534c56bd7abd352b1d1da129f2d386280b98ae7f4cf96bbe6c"} Nov 02 22:29:24.084259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.084241 5859 scope.go:122] "RemoveContainer" containerID="e3f201ed489bed5cf1087ab95d945184f9554ba18b79fda3345a37540af16431" Nov 02 22:29:24.225594 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.225524 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 22:29:24.271680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.271657 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/simulator-w1-5f765a-kserve-77d5c979d8-7gnpl"] Nov 02 22:29:24.328041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.327984 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 22:29:24.333379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.333360 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/premium-w1-5f765a-kserve-6879f74fd5-xxqwd"] Nov 02 22:29:24.571089 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.571060 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" path="/var/lib/kubelet/pods/57f246ab-a5d2-46b0-a8fa-8ff822401222/volumes" Nov 02 22:29:24.571570 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:24.571551 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" path="/var/lib/kubelet/pods/9a4997b2-60cb-42ee-b216-e4a93622b784/volumes" Nov 02 22:29:25.945427 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:25.945394 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:29:31.651054 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:31.651026 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:31.662812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:31.662789 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:32.144992 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:32.144964 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:29:34.985516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:34.985473 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 22:29:34.985937 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:34.985682 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" podUID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" containerName="maas-api" containerID="cri-o://982ac9e2e162e04d48286893e423cc8609c3aa076231f2b29b19daeef5ac2676" gracePeriod=30 Nov 02 22:29:35.156846 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.156813 5859 generic.go:356] "Generic (PLEG): container finished" podID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" containerID="982ac9e2e162e04d48286893e423cc8609c3aa076231f2b29b19daeef5ac2676" exitCode=0 Nov 02 22:29:35.157015 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.156875 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" event={"ID":"ecc52e39-8e8e-4aae-913b-1a46b6159d62","Type":"ContainerDied","Data":"982ac9e2e162e04d48286893e423cc8609c3aa076231f2b29b19daeef5ac2676"} Nov 02 22:29:35.253263 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.253206 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 22:29:35.270398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.270371 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\") pod \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " Nov 02 22:29:35.270522 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.270503 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") pod \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " Nov 02 22:29:35.270570 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.270548 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\" (UniqueName: \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\") pod \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " Nov 02 22:29:35.270616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.270604 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\") pod \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\" (UID: \"ecc52e39-8e8e-4aae-913b-1a46b6159d62\") " Nov 02 22:29:35.270696 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.270672 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca" pod "ecc52e39-8e8e-4aae-913b-1a46b6159d62" (UID: "ecc52e39-8e8e-4aae-913b-1a46b6159d62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:35.271025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.271004 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc52e39-8e8e-4aae-913b-1a46b6159d62-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.272555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.272524 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6" pod "ecc52e39-8e8e-4aae-913b-1a46b6159d62" (UID: "ecc52e39-8e8e-4aae-913b-1a46b6159d62"). InnerVolumeSpecName "kube-api-access-mj5m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:35.272683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.272659 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs" pod "ecc52e39-8e8e-4aae-913b-1a46b6159d62" (UID: "ecc52e39-8e8e-4aae-913b-1a46b6159d62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:35.272885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.272867 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls" pod "ecc52e39-8e8e-4aae-913b-1a46b6159d62" (UID: "ecc52e39-8e8e-4aae-913b-1a46b6159d62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:35.373614 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.372773 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.373614 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.372811 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj5m6\" (UniqueName: \"kubernetes.io/projected/ecc52e39-8e8e-4aae-913b-1a46b6159d62-kube-api-access-mj5m6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:35.373614 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:35.372826 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc52e39-8e8e-4aae-913b-1a46b6159d62-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:36.163515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.163482 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" Nov 02 22:29:36.163963 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.163522 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5" event={"ID":"ecc52e39-8e8e-4aae-913b-1a46b6159d62","Type":"ContainerDied","Data":"370ad2f3c2e6fffe37b5cb9c40b0888e1c6221b03b3b7387e72feaebd1d3b502"} Nov 02 22:29:36.163963 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.163580 5859 scope.go:122] "RemoveContainer" containerID="982ac9e2e162e04d48286893e423cc8609c3aa076231f2b29b19daeef5ac2676" Nov 02 22:29:36.317124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.317098 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 22:29:36.322767 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.322744 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5f765a-8674849cbb-thth5"] Nov 02 22:29:36.570364 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:36.570336 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" path="/var/lib/kubelet/pods/ecc52e39-8e8e-4aae-913b-1a46b6159d62/volumes" Nov 02 22:29:39.762337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:39.762300 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 22:29:39.762718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:39.762488 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" containerID="cri-o://a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf" gracePeriod=30 Nov 02 22:29:45.048361 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.048336 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 22:29:45.171051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171021 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171051 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171053 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171110 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171135 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171181 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171262 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171225 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171269 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171310 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171341 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\") pod \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\" (UID: \"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476\") " Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171344 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171434 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:45.171469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171457 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:45.171792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:45.171792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171709 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:45.171923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171840 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.171923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171854 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.171923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171863 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.171923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171872 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.171923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.171880 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.173535 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.173512 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:29:45.173670 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.173566 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:45.173757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.173731 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "kube-api-access-sxtwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:45.173819 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.173801 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy" pod "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" (UID: "2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:45.196751 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.196730 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 22:29:45.225470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.225446 5859 generic.go:356] "Generic (PLEG): container finished" podID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerID="a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf" exitCode=0 Nov 02 22:29:45.225592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.225496 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" Nov 02 22:29:45.225592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.225511 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerDied","Data":"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf"} Nov 02 22:29:45.225592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.225551 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4" event={"ID":"2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476","Type":"ContainerDied","Data":"fcbc420af5860b3ea44e8780d736ed2e494fc65950404e0281a419b045f2de4e"} Nov 02 22:29:45.225592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.225573 5859 scope.go:122] "RemoveContainer" containerID="a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf" Nov 02 22:29:45.235556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.235534 5859 scope.go:122] "RemoveContainer" containerID="058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6" Nov 02 22:29:45.249437 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.249416 5859 scope.go:122] "RemoveContainer" containerID="a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf" Nov 02 22:29:45.249744 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:45.249704 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf\": container with ID starting with a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf not found: ID does not exist" containerID="a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf" Nov 02 22:29:45.249892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.249764 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf"} err="failed to get container status \"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf\": rpc error: code = NotFound desc = could not find container \"a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf\": container with ID starting with a9da7ee3f4a6e345d0b9cfda39644dc4360e67a8d082b3315f19e385cb5e41cf not found: ID does not exist" Nov 02 22:29:45.249892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.249786 5859 scope.go:122] "RemoveContainer" containerID="058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6" Nov 02 22:29:45.250289 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:45.250267 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6\": container with ID starting with 058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6 not found: ID does not exist" containerID="058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6" Nov 02 22:29:45.250377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.250303 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6"} err="failed to get container status \"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6\": rpc error: code = NotFound desc = could not find container \"058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6\": container with ID starting with 058ffdb09c7d999ab3a705dd823985ea527ec5a7b13f11e347e441acbf6537f6 not found: ID does not exist" Nov 02 22:29:45.272901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.272880 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.272901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.272902 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtwj\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-kube-api-access-sxtwj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.273050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.272912 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.273050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.272921 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:45.415093 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.415069 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 22:29:45.418827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:45.418807 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-58ca7f-openshift-default-64dc8986b9-v77f4"] Nov 02 22:29:46.571597 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:46.571563 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" path="/var/lib/kubelet/pods/2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476/volumes" Nov 02 22:29:55.946758 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:29:55.946719 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:29:58.508905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:58.508866 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 22:29:58.606699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:29:58.606665 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 22:30:00.165810 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.165730 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:00.166250 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166236 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.166297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166251 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.166297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166267 5859 cpu_manager.go:414] "containerMap: removing container" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" Nov 02 22:30:00.166297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166273 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" Nov 02 22:30:00.166297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166285 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.166297 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166290 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166305 5859 cpu_manager.go:414] "containerMap: removing container" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166310 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166321 5859 cpu_manager.go:414] "containerMap: removing container" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166325 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166337 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" containerName="maas-api" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166342 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" containerName="maas-api" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166417 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166425 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ed1713-c414-44f6-9370-0fc4a301881a" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166434 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4997b2-60cb-42ee-b216-e4a93622b784" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166442 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f246ab-a5d2-46b0-a8fa-8ff822401222" containerName="main" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166448 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc52e39-8e8e-4aae-913b-1a46b6159d62" containerName="maas-api" Nov 02 22:30:00.166458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.166453 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="416c9c5e-19f0-4a38-976b-c5c74c90ed66" containerName="istio-proxy" Nov 02 22:30:00.171616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.171595 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.197721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.197696 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:00.198599 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.198575 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5bf6ea-f433-4ed6-97f7-9dcfe45d1476" containerName="istio-proxy" Nov 02 22:30:00.203018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.202992 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.217435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.217412 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" containerID="cri-o://bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1" gracePeriod=45 Nov 02 22:30:00.241681 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.241661 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:30:00.256259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.256231 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:00.270720 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.270696 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:00.312934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.312902 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.313068 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.312964 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sm5k\" (UniqueName: \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.313068 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.313056 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qln\" (UniqueName: \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.313186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.313084 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.313798 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.313776 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 02 22:30:00.414926 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.414898 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.415114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.415081 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.415182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.415124 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sm5k\" (UniqueName: \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.415182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.415142 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qln\" (UniqueName: \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.416228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.416162 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.424420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.416610 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.468727 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.468702 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qln\" (UniqueName: \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\") pod \"maas-api-key-cleanup-29894310-7kgt8\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.468869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.468847 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sm5k\" (UniqueName: \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.469538 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.469521 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 22:30:00.484671 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.484654 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:00.516611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.516586 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:00.617784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.617756 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.617945 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.617837 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.617945 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.617888 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.617945 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.617941 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\" (UniqueName: \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.618174 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618033 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.618235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618186 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\") pod \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\" (UID: \"b7a1c035-2d96-4e95-8da5-a0642d874fb5\") " Nov 02 22:30:00.618314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618296 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.618460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618433 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.618558 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618522 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.618712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618697 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.618712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618712 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.618831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.618721 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.620196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.620172 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.620298 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.620193 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:00.621359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.621339 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb" pod "b7a1c035-2d96-4e95-8da5-a0642d874fb5" (UID: "b7a1c035-2d96-4e95-8da5-a0642d874fb5"). InnerVolumeSpecName "kube-api-access-9trmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:00.720189 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.720119 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a1c035-2d96-4e95-8da5-a0642d874fb5-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.720189 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.720147 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trmb\" (UniqueName: \"kubernetes.io/projected/b7a1c035-2d96-4e95-8da5-a0642d874fb5-kube-api-access-9trmb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.720189 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.720161 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a1c035-2d96-4e95-8da5-a0642d874fb5-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.852197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.852138 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:00.857406 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:30:00.857382 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8995831_b4f1_4c0d_83bf_98b815f4ad20.slice/crio-94c4cc96b7e4f84a674b674c221af15dc63356b19237b393420dc2a5b10fba41 WatchSource:0}: Error finding container 94c4cc96b7e4f84a674b674c221af15dc63356b19237b393420dc2a5b10fba41: Status 404 returned error can't find the container with id 94c4cc96b7e4f84a674b674c221af15dc63356b19237b393420dc2a5b10fba41 Nov 02 22:30:00.904652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:00.904629 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:00.909611 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:30:00.909587 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2abac4_db3e_4ee9_8e05_485d783ecbbc.slice/crio-165445f274a5abd5a192cf13f760789ca4d18d8109d055c5b14daf91390ff278 WatchSource:0}: Error finding container 165445f274a5abd5a192cf13f760789ca4d18d8109d055c5b14daf91390ff278: Status 404 returned error can't find the container with id 165445f274a5abd5a192cf13f760789ca4d18d8109d055c5b14daf91390ff278 Nov 02 22:30:01.323757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.323732 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerStarted","Data":"c7f97de5f11a1d88cfc6827815348a4fde62cf2e22ef7423ad15cfba9160c7c3"} Nov 02 22:30:01.324149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.323780 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerStarted","Data":"94c4cc96b7e4f84a674b674c221af15dc63356b19237b393420dc2a5b10fba41"} Nov 02 22:30:01.325077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.325054 5859 generic.go:356] "Generic (PLEG): container finished" podID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerID="bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1" exitCode=0 Nov 02 22:30:01.325135 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.325103 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" Nov 02 22:30:01.325171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.325129 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" event={"ID":"b7a1c035-2d96-4e95-8da5-a0642d874fb5","Type":"ContainerDied","Data":"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1"} Nov 02 22:30:01.325171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.325161 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz" event={"ID":"b7a1c035-2d96-4e95-8da5-a0642d874fb5","Type":"ContainerDied","Data":"e9e74f3aae79bf5c7fd71b0ff3eefd61f04c33c5140e6f86bd672246c394ec30"} Nov 02 22:30:01.325243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.325179 5859 scope.go:122] "RemoveContainer" containerID="bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1" Nov 02 22:30:01.326783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.326758 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerStarted","Data":"4370a0d1d93d5b91d487d855aee5e9c88eff0bfa0a391ca69749520f559a9b62"} Nov 02 22:30:01.326896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.326802 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerStarted","Data":"165445f274a5abd5a192cf13f760789ca4d18d8109d055c5b14daf91390ff278"} Nov 02 22:30:01.336144 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.336114 5859 scope.go:122] "RemoveContainer" containerID="bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1" Nov 02 22:30:01.336365 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:01.336346 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1\": container with ID starting with bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1 not found: ID does not exist" containerID="bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1" Nov 02 22:30:01.336419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.336371 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1"} err="failed to get container status \"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1\": rpc error: code = NotFound desc = could not find container \"bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1\": container with ID starting with bf045655060140db9f1e69d8ecd3def63e08e57f9712bfaec5477c0a1b04fed1 not found: ID does not exist" Nov 02 22:30:01.429346 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.429306 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" podStartSLOduration=1.429295329 podStartE2EDuration="1.429295329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.427878196 +0000 UTC m=+4840.976127994" watchObservedRunningTime="2026-11-02 22:30:01.429295329 +0000 UTC m=+4840.977545109" Nov 02 22:30:01.531414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.531388 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 22:30:01.534888 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.534867 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-w4-d6cb38-kserve-6ff994fb9c-rzvlz"] Nov 02 22:30:01.637631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:01.637585 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" podStartSLOduration=1.637560413 podStartE2EDuration="1.637560413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.637253138 +0000 UTC m=+4841.185502948" watchObservedRunningTime="2026-11-02 22:30:01.637560413 +0000 UTC m=+4841.185810227" Nov 02 22:30:02.333996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.333932 5859 generic.go:356] "Generic (PLEG): container finished" podID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerID="4370a0d1d93d5b91d487d855aee5e9c88eff0bfa0a391ca69749520f559a9b62" exitCode=60 Nov 02 22:30:02.334428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.334022 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerDied","Data":"4370a0d1d93d5b91d487d855aee5e9c88eff0bfa0a391ca69749520f559a9b62"} Nov 02 22:30:02.334428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.334179 5859 scope.go:122] "RemoveContainer" containerID="4370a0d1d93d5b91d487d855aee5e9c88eff0bfa0a391ca69749520f559a9b62" Nov 02 22:30:02.335812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.335788 5859 generic.go:356] "Generic (PLEG): container finished" podID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerID="c7f97de5f11a1d88cfc6827815348a4fde62cf2e22ef7423ad15cfba9160c7c3" exitCode=60 Nov 02 22:30:02.335924 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.335878 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerDied","Data":"c7f97de5f11a1d88cfc6827815348a4fde62cf2e22ef7423ad15cfba9160c7c3"} Nov 02 22:30:02.336048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.336033 5859 scope.go:122] "RemoveContainer" containerID="c7f97de5f11a1d88cfc6827815348a4fde62cf2e22ef7423ad15cfba9160c7c3" Nov 02 22:30:02.570924 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:02.570898 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" path="/var/lib/kubelet/pods/b7a1c035-2d96-4e95-8da5-a0642d874fb5/volumes" Nov 02 22:30:03.344462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.344431 5859 generic.go:356] "Generic (PLEG): container finished" podID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerID="c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce" exitCode=60 Nov 02 22:30:03.344870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.344514 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerDied","Data":"c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce"} Nov 02 22:30:03.344870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.344563 5859 scope.go:122] "RemoveContainer" containerID="c7f97de5f11a1d88cfc6827815348a4fde62cf2e22ef7423ad15cfba9160c7c3" Nov 02 22:30:03.344870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.344725 5859 scope.go:122] "RemoveContainer" containerID="c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce" Nov 02 22:30:03.345033 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:03.344947 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-7kgt8_odh-ai-gateway-infra(d8995831-b4f1-4c0d-83bf-98b815f4ad20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" Nov 02 22:30:03.346370 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.346348 5859 generic.go:356] "Generic (PLEG): container finished" podID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerID="da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420" exitCode=60 Nov 02 22:30:03.346469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.346401 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerDied","Data":"da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420"} Nov 02 22:30:03.346620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.346607 5859 scope.go:122] "RemoveContainer" containerID="da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420" Nov 02 22:30:03.346805 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:03.346790 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7_odh-ai-gateway-infra(2f2abac4-db3e-4ee9-8e05-485d783ecbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" Nov 02 22:30:03.355902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.355886 5859 scope.go:122] "RemoveContainer" containerID="4370a0d1d93d5b91d487d855aee5e9c88eff0bfa0a391ca69749520f559a9b62" Nov 02 22:30:03.509213 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.509187 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" containerID="cri-o://43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29" gracePeriod=25 Nov 02 22:30:03.607510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:03.607478 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" containerID="cri-o://c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173" gracePeriod=25 Nov 02 22:30:04.354747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:04.354715 5859 scope.go:122] "RemoveContainer" containerID="c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce" Nov 02 22:30:04.355197 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:04.354997 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-7kgt8_odh-ai-gateway-infra(d8995831-b4f1-4c0d-83bf-98b815f4ad20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" Nov 02 22:30:04.356398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:04.356376 5859 scope.go:122] "RemoveContainer" containerID="da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420" Nov 02 22:30:04.356596 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:04.356578 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7_odh-ai-gateway-infra(2f2abac4-db3e-4ee9-8e05-485d783ecbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" Nov 02 22:30:08.178867 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:08.178828 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:08.179318 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:08.178828 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:13.348593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.348560 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 22:30:13.349004 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.348745 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" podUID="9fc7b595-5f5c-48cc-9297-e30e154c6053" containerName="maas-api" containerID="cri-o://e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b" gracePeriod=30 Nov 02 22:30:13.597071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.597048 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 22:30:13.742736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.742672 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\") pod \"9fc7b595-5f5c-48cc-9297-e30e154c6053\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " Nov 02 22:30:13.742736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.742717 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") pod \"9fc7b595-5f5c-48cc-9297-e30e154c6053\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " Nov 02 22:30:13.742974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.742739 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\" (UniqueName: \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\") pod \"9fc7b595-5f5c-48cc-9297-e30e154c6053\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " Nov 02 22:30:13.742974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.742836 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\") pod \"9fc7b595-5f5c-48cc-9297-e30e154c6053\" (UID: \"9fc7b595-5f5c-48cc-9297-e30e154c6053\") " Nov 02 22:30:13.743474 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.743436 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca" pod "9fc7b595-5f5c-48cc-9297-e30e154c6053" (UID: "9fc7b595-5f5c-48cc-9297-e30e154c6053"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:13.744966 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.744922 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls" pod "9fc7b595-5f5c-48cc-9297-e30e154c6053" (UID: "9fc7b595-5f5c-48cc-9297-e30e154c6053"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:13.744966 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.744939 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs" pod "9fc7b595-5f5c-48cc-9297-e30e154c6053" (UID: "9fc7b595-5f5c-48cc-9297-e30e154c6053"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:13.745097 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.745010 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd" pod "9fc7b595-5f5c-48cc-9297-e30e154c6053" (UID: "9fc7b595-5f5c-48cc-9297-e30e154c6053"). InnerVolumeSpecName "kube-api-access-wgvmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:13.843896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.843870 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:13.844039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.843901 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fc7b595-5f5c-48cc-9297-e30e154c6053-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:13.844039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.843917 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvmd\" (UniqueName: \"kubernetes.io/projected/9fc7b595-5f5c-48cc-9297-e30e154c6053-kube-api-access-wgvmd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:13.844039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:13.843930 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7b595-5f5c-48cc-9297-e30e154c6053-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:14.423511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.423478 5859 generic.go:356] "Generic (PLEG): container finished" podID="9fc7b595-5f5c-48cc-9297-e30e154c6053" containerID="e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b" exitCode=0 Nov 02 22:30:14.423929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.423513 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" event={"ID":"9fc7b595-5f5c-48cc-9297-e30e154c6053","Type":"ContainerDied","Data":"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b"} Nov 02 22:30:14.423929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.423565 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" event={"ID":"9fc7b595-5f5c-48cc-9297-e30e154c6053","Type":"ContainerDied","Data":"85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c"} Nov 02 22:30:14.423929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.423588 5859 scope.go:122] "RemoveContainer" containerID="e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b" Nov 02 22:30:14.423929 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.423528 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms" Nov 02 22:30:14.441480 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:14.441446 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7b595_5f5c_48cc_9297_e30e154c6053.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7b595_5f5c_48cc_9297_e30e154c6053.slice/crio-85ca53cfae76f08499617743e717c03918845514ff31471dbe6dddf9b42d997c\": RecentStats: unable to find data in memory cache]" Nov 02 22:30:14.444853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.444830 5859 scope.go:122] "RemoveContainer" containerID="e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b" Nov 02 22:30:14.445158 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:14.445125 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b\": container with ID starting with e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b not found: ID does not exist" containerID="e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b" Nov 02 22:30:14.445308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.445164 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b"} err="failed to get container status \"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b\": rpc error: code = NotFound desc = could not find container \"e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b\": container with ID starting with e76635c3640505cb32f2b71962b6ce7096518f85d5eb760850badf5f4865812b not found: ID does not exist" Nov 02 22:30:14.563930 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.563913 5859 scope.go:122] "RemoveContainer" containerID="c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce" Nov 02 22:30:14.577509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.577487 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 22:30:14.624020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:14.623995 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d6cb38-77db98f9f-rr2ms"] Nov 02 22:30:15.432345 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:15.432315 5859 generic.go:356] "Generic (PLEG): container finished" podID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerID="99312b699e692e141028460347a5b8325676b06990864d543cd7e1f14a93a29e" exitCode=60 Nov 02 22:30:15.432742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:15.432368 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerDied","Data":"99312b699e692e141028460347a5b8325676b06990864d543cd7e1f14a93a29e"} Nov 02 22:30:15.432742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:15.432427 5859 scope.go:122] "RemoveContainer" containerID="c3d97cff0339a307298604c2a73a2997f1f138b53e39a3acb77cc63787aff8ce" Nov 02 22:30:15.432855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:15.432748 5859 scope.go:122] "RemoveContainer" containerID="99312b699e692e141028460347a5b8325676b06990864d543cd7e1f14a93a29e" Nov 02 22:30:15.433072 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:15.433046 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894310-7kgt8_odh-ai-gateway-infra(d8995831-b4f1-4c0d-83bf-98b815f4ad20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" Nov 02 22:30:16.549226 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.549198 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:16.570590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.570558 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc7b595-5f5c-48cc-9297-e30e154c6053" path="/var/lib/kubelet/pods/9fc7b595-5f5c-48cc-9297-e30e154c6053/volumes" Nov 02 22:30:16.680658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.680636 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:16.872691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.872661 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\" (UniqueName: \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\") pod \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " Nov 02 22:30:16.872853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.872734 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\") pod \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\" (UID: \"d8995831-b4f1-4c0d-83bf-98b815f4ad20\") " Nov 02 22:30:16.873115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.873093 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca" pod "d8995831-b4f1-4c0d-83bf-98b815f4ad20" (UID: "d8995831-b4f1-4c0d-83bf-98b815f4ad20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:16.874738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.874713 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln" pod "d8995831-b4f1-4c0d-83bf-98b815f4ad20" (UID: "d8995831-b4f1-4c0d-83bf-98b815f4ad20"). InnerVolumeSpecName "kube-api-access-z6qln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:16.974177 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.974155 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6qln\" (UniqueName: \"kubernetes.io/projected/d8995831-b4f1-4c0d-83bf-98b815f4ad20-kube-api-access-z6qln\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:16.974177 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:16.974177 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8995831-b4f1-4c0d-83bf-98b815f4ad20-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:17.447434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.447408 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" Nov 02 22:30:17.447434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.447430 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8" event={"ID":"d8995831-b4f1-4c0d-83bf-98b815f4ad20","Type":"ContainerDied","Data":"94c4cc96b7e4f84a674b674c221af15dc63356b19237b393420dc2a5b10fba41"} Nov 02 22:30:17.447698 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.447482 5859 scope.go:122] "RemoveContainer" containerID="99312b699e692e141028460347a5b8325676b06990864d543cd7e1f14a93a29e" Nov 02 22:30:17.563547 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.563520 5859 scope.go:122] "RemoveContainer" containerID="da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420" Nov 02 22:30:17.600655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.600626 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:17.647045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:17.647023 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894310-7kgt8"] Nov 02 22:30:18.178601 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.178531 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:18.178601 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.178550 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:18.456223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.456171 5859 generic.go:356] "Generic (PLEG): container finished" podID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerID="1f2674215472fb08eb9825950d6e3594639b92e474d0ab5057a2f1f3c3305ef3" exitCode=60 Nov 02 22:30:18.456329 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.456243 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerDied","Data":"1f2674215472fb08eb9825950d6e3594639b92e474d0ab5057a2f1f3c3305ef3"} Nov 02 22:30:18.456329 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.456289 5859 scope.go:122] "RemoveContainer" containerID="da72503df4d80bda44779e4d6976e82320590ca7325719d1f15ef580199a8420" Nov 02 22:30:18.456600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.456581 5859 scope.go:122] "RemoveContainer" containerID="1f2674215472fb08eb9825950d6e3594639b92e474d0ab5057a2f1f3c3305ef3" Nov 02 22:30:18.456826 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:18.456808 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7_odh-ai-gateway-infra(2f2abac4-db3e-4ee9-8e05-485d783ecbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" Nov 02 22:30:18.570309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:18.570285 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" path="/var/lib/kubelet/pods/d8995831-b4f1-4c0d-83bf-98b815f4ad20/volumes" Nov 02 22:30:19.571040 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.571007 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:19.716294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.716263 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:19.802703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.802674 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\") pod \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " Nov 02 22:30:19.802849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.802803 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\" (UniqueName: \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\") pod \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\" (UID: \"2f2abac4-db3e-4ee9-8e05-485d783ecbbc\") " Nov 02 22:30:19.803055 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.803033 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca" pod "2f2abac4-db3e-4ee9-8e05-485d783ecbbc" (UID: "2f2abac4-db3e-4ee9-8e05-485d783ecbbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:19.803135 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.803097 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:19.804728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.804708 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k" pod "2f2abac4-db3e-4ee9-8e05-485d783ecbbc" (UID: "2f2abac4-db3e-4ee9-8e05-485d783ecbbc"). InnerVolumeSpecName "kube-api-access-4sm5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:19.904610 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:19.904555 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sm5k\" (UniqueName: \"kubernetes.io/projected/2f2abac4-db3e-4ee9-8e05-485d783ecbbc-kube-api-access-4sm5k\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:20.478025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:20.477998 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" Nov 02 22:30:20.478195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:20.477997 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7" event={"ID":"2f2abac4-db3e-4ee9-8e05-485d783ecbbc","Type":"ContainerDied","Data":"165445f274a5abd5a192cf13f760789ca4d18d8109d055c5b14daf91390ff278"} Nov 02 22:30:20.478195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:20.478135 5859 scope.go:122] "RemoveContainer" containerID="1f2674215472fb08eb9825950d6e3594639b92e474d0ab5057a2f1f3c3305ef3" Nov 02 22:30:20.631748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:20.631720 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:20.635081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:20.635053 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-97995b-29894310-89vd7"] Nov 02 22:30:21.736414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:21.736383 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 22:30:22.570620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:22.570589 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" path="/var/lib/kubelet/pods/2f2abac4-db3e-4ee9-8e05-485d783ecbbc/volumes" Nov 02 22:30:23.772335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.772275 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 22:30:23.840203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.840175 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\") pod \"b295cb13-d8bb-42e7-ac51-c87718503401\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " Nov 02 22:30:23.840349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.840225 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\" (UniqueName: \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\") pod \"b295cb13-d8bb-42e7-ac51-c87718503401\" (UID: \"b295cb13-d8bb-42e7-ac51-c87718503401\") " Nov 02 22:30:23.840536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.840509 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume" pod "b295cb13-d8bb-42e7-ac51-c87718503401" (UID: "b295cb13-d8bb-42e7-ac51-c87718503401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:23.840650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.840636 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b295cb13-d8bb-42e7-ac51-c87718503401-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.842410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.842387 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv" pod "b295cb13-d8bb-42e7-ac51-c87718503401" (UID: "b295cb13-d8bb-42e7-ac51-c87718503401"). InnerVolumeSpecName "kube-api-access-267hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:23.862983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.862960 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 22:30:23.941875 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.941850 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\" (UniqueName: \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\") pod \"d3eca70f-a55d-40a6-9e01-1504eddb3704\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " Nov 02 22:30:23.942030 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.941915 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\") pod \"d3eca70f-a55d-40a6-9e01-1504eddb3704\" (UID: \"d3eca70f-a55d-40a6-9e01-1504eddb3704\") " Nov 02 22:30:23.942203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.942186 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-267hv\" (UniqueName: \"kubernetes.io/projected/b295cb13-d8bb-42e7-ac51-c87718503401-kube-api-access-267hv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:23.942335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.942313 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume" pod "d3eca70f-a55d-40a6-9e01-1504eddb3704" (UID: "d3eca70f-a55d-40a6-9e01-1504eddb3704"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:23.943732 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:23.943713 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8" pod "d3eca70f-a55d-40a6-9e01-1504eddb3704" (UID: "d3eca70f-a55d-40a6-9e01-1504eddb3704"). InnerVolumeSpecName "kube-api-access-bmfk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:24.043501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.043432 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3eca70f-a55d-40a6-9e01-1504eddb3704-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:24.043501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.043460 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfk8\" (UniqueName: \"kubernetes.io/projected/d3eca70f-a55d-40a6-9e01-1504eddb3704-kube-api-access-bmfk8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:24.502786 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.502753 5859 generic.go:356] "Generic (PLEG): container finished" podID="b295cb13-d8bb-42e7-ac51-c87718503401" containerID="43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29" exitCode=0 Nov 02 22:30:24.502962 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.502795 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" Nov 02 22:30:24.502962 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.502845 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" event={"ID":"b295cb13-d8bb-42e7-ac51-c87718503401","Type":"ContainerDied","Data":"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29"} Nov 02 22:30:24.502962 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.502914 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj" event={"ID":"b295cb13-d8bb-42e7-ac51-c87718503401","Type":"ContainerDied","Data":"fa866a16bcae33c7aebb286dc56dc6bc8a285ebe86abc9278cbd88fb3d9aa525"} Nov 02 22:30:24.502962 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.502937 5859 scope.go:122] "RemoveContainer" containerID="43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29" Nov 02 22:30:24.504223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.504199 5859 generic.go:356] "Generic (PLEG): container finished" podID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerID="c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173" exitCode=0 Nov 02 22:30:24.504337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.504237 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" event={"ID":"d3eca70f-a55d-40a6-9e01-1504eddb3704","Type":"ContainerDied","Data":"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173"} Nov 02 22:30:24.504337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.504246 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" Nov 02 22:30:24.504337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.504273 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl" event={"ID":"d3eca70f-a55d-40a6-9e01-1504eddb3704","Type":"ContainerDied","Data":"9d5619a44853ff35a71a24abd70d45a7043a76eda639dd2f83ae826023d2d425"} Nov 02 22:30:24.515078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.513487 5859 scope.go:122] "RemoveContainer" containerID="43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29" Nov 02 22:30:24.515078 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:24.513806 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29\": container with ID starting with 43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29 not found: ID does not exist" containerID="43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29" Nov 02 22:30:24.515078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.513837 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29"} err="failed to get container status \"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29\": rpc error: code = NotFound desc = could not find container \"43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29\": container with ID starting with 43ff8276df75efae25cce6e796acddb0d31a2ac374054a8a4adb60e1f5faac29 not found: ID does not exist" Nov 02 22:30:24.515078 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.513860 5859 scope.go:122] "RemoveContainer" containerID="c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173" Nov 02 22:30:24.523017 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.522946 5859 scope.go:122] "RemoveContainer" containerID="c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173" Nov 02 22:30:24.523261 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:24.523246 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173\": container with ID starting with c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173 not found: ID does not exist" containerID="c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173" Nov 02 22:30:24.523316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.523266 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173"} err="failed to get container status \"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173\": rpc error: code = NotFound desc = could not find container \"c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173\": container with ID starting with c25afbd384978ac56900b9b9dd2ebc44a2b703e10097d951a3b846a593a8d173 not found: ID does not exist" Nov 02 22:30:24.654799 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.654770 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 22:30:24.701193 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.701168 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d6cb38-78c5455bcf-wc8gl"] Nov 02 22:30:24.803372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.803303 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 22:30:24.805656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:24.805634 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-d6cb38-74f947676c-wjjgj"] Nov 02 22:30:25.943916 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:25.943883 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:30:26.043409 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:26.043380 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 22:30:26.043590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:26.043566 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" containerID="cri-o://9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c" gracePeriod=30 Nov 02 22:30:26.569033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:26.568999 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" path="/var/lib/kubelet/pods/b295cb13-d8bb-42e7-ac51-c87718503401/volumes" Nov 02 22:30:26.569382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:26.569369 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" path="/var/lib/kubelet/pods/d3eca70f-a55d-40a6-9e01-1504eddb3704/volumes" Nov 02 22:30:26.737143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:26.737110 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" containerID="cri-o://890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4" gracePeriod=25 Nov 02 22:30:29.088334 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:29.088299 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:31.115757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.115731 5859 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:30:31.116081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.115779 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:30:31.303275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.303251 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 22:30:31.409788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409755 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409806 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409854 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409891 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409917 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409935 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.409998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.409979 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.410359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410042 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.410359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410086 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.410359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410108 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\") pod \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\" (UID: \"bff8d3b8-b702-432d-8965-b5fc4c3072ef\") " Nov 02 22:30:31.410359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410224 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.410567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410376 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.410567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410407 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:31.410679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410581 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.410736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410714 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.410794 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410735 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.410794 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410751 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.410794 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410765 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.410794 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.410779 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.412056 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.412031 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:31.412238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.412221 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:31.412304 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.412239 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "kube-api-access-7f5wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:31.412304 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.412272 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token" pod "bff8d3b8-b702-432d-8965-b5fc4c3072ef" (UID: "bff8d3b8-b702-432d-8965-b5fc4c3072ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:31.511833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.511805 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5wp\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-kube-api-access-7f5wp\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.511833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.511829 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.512015 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.511838 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.512015 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.511848 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bff8d3b8-b702-432d-8965-b5fc4c3072ef-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.552862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.552835 5859 generic.go:356] "Generic (PLEG): container finished" podID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerID="9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c" exitCode=0 Nov 02 22:30:31.553008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.552883 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" Nov 02 22:30:31.553008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.552909 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" event={"ID":"bff8d3b8-b702-432d-8965-b5fc4c3072ef","Type":"ContainerDied","Data":"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c"} Nov 02 22:30:31.553008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.552984 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc" event={"ID":"bff8d3b8-b702-432d-8965-b5fc4c3072ef","Type":"ContainerDied","Data":"86545492711e34c3b5f8a8ab3b1bc62d033d6b3c623998ee4f82b72c588fe474"} Nov 02 22:30:31.553171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.553012 5859 scope.go:122] "RemoveContainer" containerID="9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c" Nov 02 22:30:31.562441 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.562425 5859 scope.go:122] "RemoveContainer" containerID="9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c" Nov 02 22:30:31.562705 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:31.562685 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c\": container with ID starting with 9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c not found: ID does not exist" containerID="9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c" Nov 02 22:30:31.562782 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.562713 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c"} err="failed to get container status \"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c\": rpc error: code = NotFound desc = could not find container \"9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c\": container with ID starting with 9ea16689ca9b082cb9a081742a2485145592fcd9d15d73eafca63a639dd7f43c not found: ID does not exist" Nov 02 22:30:31.709586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.709509 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 22:30:31.715313 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:31.715289 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-pv2bc"] Nov 02 22:30:32.569646 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:32.569616 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" path="/var/lib/kubelet/pods/bff8d3b8-b702-432d-8965-b5fc4c3072ef/volumes" Nov 02 22:30:34.896365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:34.896333 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 22:30:34.896796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:34.896520 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" containerID="cri-o://5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea" gracePeriod=30 Nov 02 22:30:39.088561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:39.088525 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:30:40.147853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.147829 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 22:30:40.289343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289283 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289328 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289347 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289428 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289465 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289499 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289542 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289581 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289657 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\") pod \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\" (UID: \"803259bc-9cdf-4d90-affb-b16e1b3b8f0f\") " Nov 02 22:30:40.289905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289724 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:40.289905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.289758 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:40.290145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.290053 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.290145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.290079 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.290145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.290084 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:40.290145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.290109 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:40.290407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.290150 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:40.291484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.291459 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:40.291484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.291475 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:40.291607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.291558 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "kube-api-access-c2xdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:40.291747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.291730 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token" pod "803259bc-9cdf-4d90-affb-b16e1b3b8f0f" (UID: "803259bc-9cdf-4d90-affb-b16e1b3b8f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392310 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392340 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392357 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2xdm\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-kube-api-access-c2xdm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392371 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392389 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392404 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.399970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.392417 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803259bc-9cdf-4d90-affb-b16e1b3b8f0f-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.600644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.600613 5859 generic.go:356] "Generic (PLEG): container finished" podID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerID="5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea" exitCode=0 Nov 02 22:30:40.600740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.600663 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" Nov 02 22:30:40.600740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.600687 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerDied","Data":"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea"} Nov 02 22:30:40.600740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.600733 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9" event={"ID":"803259bc-9cdf-4d90-affb-b16e1b3b8f0f","Type":"ContainerDied","Data":"9c29a6ea138df712baeaadb4b46c0b849009ae00a6a93e3a3ed82f61e3fa07b0"} Nov 02 22:30:40.600842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.600761 5859 scope.go:122] "RemoveContainer" containerID="5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea" Nov 02 22:30:40.609658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.609642 5859 scope.go:122] "RemoveContainer" containerID="9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0" Nov 02 22:30:40.617607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.617591 5859 scope.go:122] "RemoveContainer" containerID="5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea" Nov 02 22:30:40.617869 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:40.617851 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea\": container with ID starting with 5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea not found: ID does not exist" containerID="5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea" Nov 02 22:30:40.617918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.617877 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea"} err="failed to get container status \"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea\": rpc error: code = NotFound desc = could not find container \"5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea\": container with ID starting with 5611b3def7a39203a72322e2c5784e4b955c0fb1eb291b7abd7a73e203fbdbea not found: ID does not exist" Nov 02 22:30:40.617918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.617893 5859 scope.go:122] "RemoveContainer" containerID="9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0" Nov 02 22:30:40.618118 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:40.618100 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0\": container with ID starting with 9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0 not found: ID does not exist" containerID="9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0" Nov 02 22:30:40.618166 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.618122 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0"} err="failed to get container status \"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0\": rpc error: code = NotFound desc = could not find container \"9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0\": container with ID starting with 9e950512c42c637bb04d00cefa0c639ee876f5e676baa1f58e8dfc4db0c528f0 not found: ID does not exist" Nov 02 22:30:40.755940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.755915 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 22:30:40.758637 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:40.758616 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6e535d-openshift-default-5ff6f7f7dc-f4wx9"] Nov 02 22:30:42.274679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.274635 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 22:30:42.275165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.274883 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" podUID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" containerName="maas-api" containerID="cri-o://fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680" gracePeriod=30 Nov 02 22:30:42.526563 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.526506 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 22:30:42.570556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.570530 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" path="/var/lib/kubelet/pods/803259bc-9cdf-4d90-affb-b16e1b3b8f0f/volumes" Nov 02 22:30:42.610156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.610131 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\") pod \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " Nov 02 22:30:42.610285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.610176 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") pod \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " Nov 02 22:30:42.610393 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.610375 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\") pod \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " Nov 02 22:30:42.610469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.610448 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\" (UniqueName: \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\") pod \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\" (UID: \"ff48d4fe-28a8-48d7-9290-18bdd2483ea9\") " Nov 02 22:30:42.610843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.610815 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca" pod "ff48d4fe-28a8-48d7-9290-18bdd2483ea9" (UID: "ff48d4fe-28a8-48d7-9290-18bdd2483ea9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:42.612516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.612473 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls" pod "ff48d4fe-28a8-48d7-9290-18bdd2483ea9" (UID: "ff48d4fe-28a8-48d7-9290-18bdd2483ea9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:42.612969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.612922 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk" pod "ff48d4fe-28a8-48d7-9290-18bdd2483ea9" (UID: "ff48d4fe-28a8-48d7-9290-18bdd2483ea9"). InnerVolumeSpecName "kube-api-access-7kddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:42.613092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.613069 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs" pod "ff48d4fe-28a8-48d7-9290-18bdd2483ea9" (UID: "ff48d4fe-28a8-48d7-9290-18bdd2483ea9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:42.614612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.614592 5859 generic.go:356] "Generic (PLEG): container finished" podID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" containerID="fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680" exitCode=0 Nov 02 22:30:42.614715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.614649 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" event={"ID":"ff48d4fe-28a8-48d7-9290-18bdd2483ea9","Type":"ContainerDied","Data":"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680"} Nov 02 22:30:42.614715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.614679 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" event={"ID":"ff48d4fe-28a8-48d7-9290-18bdd2483ea9","Type":"ContainerDied","Data":"8b610f4b20fd717bd28ae374f0ba4b4a77674e4d2cf4fbf15b1c0d0c6331dac0"} Nov 02 22:30:42.614715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.614698 5859 scope.go:122] "RemoveContainer" containerID="fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680" Nov 02 22:30:42.614887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.614787 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h" Nov 02 22:30:42.635298 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.635274 5859 scope.go:122] "RemoveContainer" containerID="fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680" Nov 02 22:30:42.635571 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:42.635551 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680\": container with ID starting with fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680 not found: ID does not exist" containerID="fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680" Nov 02 22:30:42.635641 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.635579 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680"} err="failed to get container status \"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680\": rpc error: code = NotFound desc = could not find container \"fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680\": container with ID starting with fb0d398a49d683a1baaf565d0684bbca46f820a17700397e874316cc3ed81680 not found: ID does not exist" Nov 02 22:30:42.712461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.712431 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:42.712461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.712459 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:42.712461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.712468 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:42.712719 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.712481 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kddk\" (UniqueName: \"kubernetes.io/projected/ff48d4fe-28a8-48d7-9290-18bdd2483ea9-kube-api-access-7kddk\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:42.751269 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.751240 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 22:30:42.755464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:42.755441 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0f0874-646b5dfd6b-ppg9h"] Nov 02 22:30:44.570554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:44.570523 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" path="/var/lib/kubelet/pods/ff48d4fe-28a8-48d7-9290-18bdd2483ea9/volumes" Nov 02 22:30:46.990327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:46.990303 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 22:30:47.150758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.150729 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\") pod \"82e5796c-0890-40f2-91b0-19f8669d0524\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " Nov 02 22:30:47.150908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.150857 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\" (UniqueName: \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\") pod \"82e5796c-0890-40f2-91b0-19f8669d0524\" (UID: \"82e5796c-0890-40f2-91b0-19f8669d0524\") " Nov 02 22:30:47.151107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.151085 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume" pod "82e5796c-0890-40f2-91b0-19f8669d0524" (UID: "82e5796c-0890-40f2-91b0-19f8669d0524"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:47.151290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.151277 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82e5796c-0890-40f2-91b0-19f8669d0524-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:47.152826 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.152808 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8" pod "82e5796c-0890-40f2-91b0-19f8669d0524" (UID: "82e5796c-0890-40f2-91b0-19f8669d0524"). InnerVolumeSpecName "kube-api-access-v82p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:47.252592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.252571 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82p8\" (UniqueName: \"kubernetes.io/projected/82e5796c-0890-40f2-91b0-19f8669d0524-kube-api-access-v82p8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:47.643455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.643420 5859 generic.go:356] "Generic (PLEG): container finished" podID="82e5796c-0890-40f2-91b0-19f8669d0524" containerID="890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4" exitCode=0 Nov 02 22:30:47.643631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.643469 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" Nov 02 22:30:47.643631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.643506 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" event={"ID":"82e5796c-0890-40f2-91b0-19f8669d0524","Type":"ContainerDied","Data":"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4"} Nov 02 22:30:47.643631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.643559 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh" event={"ID":"82e5796c-0890-40f2-91b0-19f8669d0524","Type":"ContainerDied","Data":"c23bfd3bc80a4c7310253de263a025c71b9d417347fb03d0663def306b66e08c"} Nov 02 22:30:47.643631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.643588 5859 scope.go:122] "RemoveContainer" containerID="890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4" Nov 02 22:30:47.653533 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.653516 5859 scope.go:122] "RemoveContainer" containerID="890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4" Nov 02 22:30:47.653831 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:47.653809 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4\": container with ID starting with 890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4 not found: ID does not exist" containerID="890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4" Nov 02 22:30:47.653897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.653843 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4"} err="failed to get container status \"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4\": rpc error: code = NotFound desc = could not find container \"890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4\": container with ID starting with 890fb82e29ca76f1b48e3bea85d9b3bf23bc4229de88b6ce3e07b173487ca6b4 not found: ID does not exist" Nov 02 22:30:47.795967 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.795920 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 22:30:47.841821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:47.841778 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6e535d-7969ddcdd7-gm6qh"] Nov 02 22:30:48.569161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:30:48.569131 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" path="/var/lib/kubelet/pods/82e5796c-0890-40f2-91b0-19f8669d0524/volumes" Nov 02 22:30:55.944429 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:30:55.944389 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:31:25.945243 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:31:25.945200 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:31:34.653879 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:34.653843 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 22:31:34.751235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:34.751203 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 22:31:39.655017 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:39.654975 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" containerID="cri-o://ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4" gracePeriod=25 Nov 02 22:31:39.751504 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:39.751475 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" containerID="cri-o://ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f" gracePeriod=25 Nov 02 22:31:41.215486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:41.215448 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:41.215983 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:41.215448 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:51.215719 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:51.215680 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:51.216119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:51.215738 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:31:55.945043 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:31:55.945001 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:31:56.412591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.412552 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 22:31:56.412848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.412788 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" podUID="e13a11b4-7cc5-40fc-8183-27323975ce1e" containerName="maas-api" containerID="cri-o://2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a" gracePeriod=30 Nov 02 22:31:56.670739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.670669 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 22:31:56.692690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.692664 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\") pod \"e13a11b4-7cc5-40fc-8183-27323975ce1e\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " Nov 02 22:31:56.692825 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.692694 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") pod \"e13a11b4-7cc5-40fc-8183-27323975ce1e\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " Nov 02 22:31:56.692825 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.692762 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\" (UniqueName: \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\") pod \"e13a11b4-7cc5-40fc-8183-27323975ce1e\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " Nov 02 22:31:56.692825 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.692820 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\") pod \"e13a11b4-7cc5-40fc-8183-27323975ce1e\" (UID: \"e13a11b4-7cc5-40fc-8183-27323975ce1e\") " Nov 02 22:31:56.693276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.693248 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca" pod "e13a11b4-7cc5-40fc-8183-27323975ce1e" (UID: "e13a11b4-7cc5-40fc-8183-27323975ce1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:56.694795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.694762 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs" pod "e13a11b4-7cc5-40fc-8183-27323975ce1e" (UID: "e13a11b4-7cc5-40fc-8183-27323975ce1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:56.694877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.694830 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5" pod "e13a11b4-7cc5-40fc-8183-27323975ce1e" (UID: "e13a11b4-7cc5-40fc-8183-27323975ce1e"). InnerVolumeSpecName "kube-api-access-4fss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:56.694877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.694826 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls" pod "e13a11b4-7cc5-40fc-8183-27323975ce1e" (UID: "e13a11b4-7cc5-40fc-8183-27323975ce1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:56.794022 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.793994 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:56.794022 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.794017 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13a11b4-7cc5-40fc-8183-27323975ce1e-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:56.794022 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.794027 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fss5\" (UniqueName: \"kubernetes.io/projected/e13a11b4-7cc5-40fc-8183-27323975ce1e-kube-api-access-4fss5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:56.794245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:56.794036 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13a11b4-7cc5-40fc-8183-27323975ce1e-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:57.024802 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.024714 5859 generic.go:356] "Generic (PLEG): container finished" podID="e13a11b4-7cc5-40fc-8183-27323975ce1e" containerID="2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a" exitCode=0 Nov 02 22:31:57.024802 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.024777 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" Nov 02 22:31:57.025322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.024795 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" event={"ID":"e13a11b4-7cc5-40fc-8183-27323975ce1e","Type":"ContainerDied","Data":"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a"} Nov 02 22:31:57.025322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.024869 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr" event={"ID":"e13a11b4-7cc5-40fc-8183-27323975ce1e","Type":"ContainerDied","Data":"8c838df68fb25e7197acb4a208a0f30842ba9ab74f46e0669bbe5a11148434fe"} Nov 02 22:31:57.025322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.024890 5859 scope.go:122] "RemoveContainer" containerID="2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a" Nov 02 22:31:57.034759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.034743 5859 scope.go:122] "RemoveContainer" containerID="2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a" Nov 02 22:31:57.035041 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:31:57.035022 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a\": container with ID starting with 2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a not found: ID does not exist" containerID="2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a" Nov 02 22:31:57.035109 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.035050 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a"} err="failed to get container status \"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a\": rpc error: code = NotFound desc = could not find container \"2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a\": container with ID starting with 2313da8f6c01fff210ea98d792582a1802d26af40901174a4529140296d82e7a not found: ID does not exist" Nov 02 22:31:57.176543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.176513 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 22:31:57.180291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:57.180271 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0c9014-78586849c4-74bmr"] Nov 02 22:31:58.569997 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:58.569947 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13a11b4-7cc5-40fc-8183-27323975ce1e" path="/var/lib/kubelet/pods/e13a11b4-7cc5-40fc-8183-27323975ce1e/volumes" Nov 02 22:31:59.895353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.895328 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 22:31:59.923540 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.923509 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\" (UniqueName: \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\") pod \"1801c51a-c2ac-441f-a8f0-316c35aec026\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " Nov 02 22:31:59.923660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.923635 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\") pod \"1801c51a-c2ac-441f-a8f0-316c35aec026\" (UID: \"1801c51a-c2ac-441f-a8f0-316c35aec026\") " Nov 02 22:31:59.924350 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.924321 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume" pod "1801c51a-c2ac-441f-a8f0-316c35aec026" (UID: "1801c51a-c2ac-441f-a8f0-316c35aec026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:59.926437 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.926397 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b" pod "1801c51a-c2ac-441f-a8f0-316c35aec026" (UID: "1801c51a-c2ac-441f-a8f0-316c35aec026"). InnerVolumeSpecName "kube-api-access-jl54b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:59.985229 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:31:59.985210 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 22:32:00.025255 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.025231 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\" (UniqueName: \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\") pod \"ffd53e7b-339d-4433-900b-141ea31a8b8a\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " Nov 02 22:32:00.025433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.025417 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\") pod \"ffd53e7b-339d-4433-900b-141ea31a8b8a\" (UID: \"ffd53e7b-339d-4433-900b-141ea31a8b8a\") " Nov 02 22:32:00.025725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.025696 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1801c51a-c2ac-441f-a8f0-316c35aec026-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:00.025725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.025725 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl54b\" (UniqueName: \"kubernetes.io/projected/1801c51a-c2ac-441f-a8f0-316c35aec026-kube-api-access-jl54b\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:00.025905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.025740 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume" pod "ffd53e7b-339d-4433-900b-141ea31a8b8a" (UID: "ffd53e7b-339d-4433-900b-141ea31a8b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:00.027338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.027305 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv" pod "ffd53e7b-339d-4433-900b-141ea31a8b8a" (UID: "ffd53e7b-339d-4433-900b-141ea31a8b8a"). InnerVolumeSpecName "kube-api-access-24tzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:00.044619 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.044598 5859 generic.go:356] "Generic (PLEG): container finished" podID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerID="ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f" exitCode=0 Nov 02 22:32:00.044716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.044641 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" Nov 02 22:32:00.044716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.044684 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" event={"ID":"ffd53e7b-339d-4433-900b-141ea31a8b8a","Type":"ContainerDied","Data":"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f"} Nov 02 22:32:00.044812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.044736 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g" event={"ID":"ffd53e7b-339d-4433-900b-141ea31a8b8a","Type":"ContainerDied","Data":"aa5d172c63cc352db3d3429e7b57a2b06363b37cf03f2ed6c487b0335a20a229"} Nov 02 22:32:00.044812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.044766 5859 scope.go:122] "RemoveContainer" containerID="ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f" Nov 02 22:32:00.046030 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.046009 5859 generic.go:356] "Generic (PLEG): container finished" podID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerID="ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4" exitCode=0 Nov 02 22:32:00.046127 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.046037 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" event={"ID":"1801c51a-c2ac-441f-a8f0-316c35aec026","Type":"ContainerDied","Data":"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4"} Nov 02 22:32:00.046127 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.046071 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" Nov 02 22:32:00.046127 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.046085 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb" event={"ID":"1801c51a-c2ac-441f-a8f0-316c35aec026","Type":"ContainerDied","Data":"0fad1b8188aa6db993a9f42c8e70ab2b1082011c9148664074ca97828ccc9b58"} Nov 02 22:32:00.054281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.054246 5859 scope.go:122] "RemoveContainer" containerID="ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f" Nov 02 22:32:00.054523 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:00.054504 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f\": container with ID starting with ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f not found: ID does not exist" containerID="ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f" Nov 02 22:32:00.054581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.054531 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f"} err="failed to get container status \"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f\": rpc error: code = NotFound desc = could not find container \"ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f\": container with ID starting with ef3f79321804d7ef94c7e2e45d7155fb2fcca0c30688506c0068a9868586de0f not found: ID does not exist" Nov 02 22:32:00.054581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.054547 5859 scope.go:122] "RemoveContainer" containerID="ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4" Nov 02 22:32:00.064997 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.064979 5859 scope.go:122] "RemoveContainer" containerID="ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4" Nov 02 22:32:00.065256 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:00.065239 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4\": container with ID starting with ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4 not found: ID does not exist" containerID="ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4" Nov 02 22:32:00.065301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.065263 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4"} err="failed to get container status \"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4\": rpc error: code = NotFound desc = could not find container \"ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4\": container with ID starting with ae034810f49bc5929f5815f0971303a9ecdc108382ec599c9415fe18fcf931b4 not found: ID does not exist" Nov 02 22:32:00.127417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.127391 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffd53e7b-339d-4433-900b-141ea31a8b8a-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:00.127509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.127426 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24tzv\" (UniqueName: \"kubernetes.io/projected/ffd53e7b-339d-4433-900b-141ea31a8b8a-kube-api-access-24tzv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:00.214857 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.214803 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 22:32:00.219199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.219178 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0c9014-698f6d657-mpgwb"] Nov 02 22:32:00.319976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.319942 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 22:32:00.323036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.323016 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0c9014-5576db4f5-57b6g"] Nov 02 22:32:00.569849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.569824 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" path="/var/lib/kubelet/pods/1801c51a-c2ac-441f-a8f0-316c35aec026/volumes" Nov 02 22:32:00.570195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:00.570181 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" path="/var/lib/kubelet/pods/ffd53e7b-339d-4433-900b-141ea31a8b8a/volumes" Nov 02 22:32:08.393416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:08.393338 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 22:32:08.393885 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:08.393535 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" containerID="cri-o://d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1" gracePeriod=30 Nov 02 22:32:11.162764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:11.162720 5859 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:32:11.163151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:11.162777 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:32:13.637969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.637925 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 22:32:13.746919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.746854 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.746919 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.746911 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.746946 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.746996 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747037 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747067 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747103 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747149 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747541 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747210 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\") pod \"dcffb9cf-bd01-4a48-9972-eaf53413e548\" (UID: \"dcffb9cf-bd01-4a48-9972-eaf53413e548\") " Nov 02 22:32:13.747541 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747267 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:13.747541 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747296 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:13.747541 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747490 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:13.747757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747567 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcffb9cf-bd01-4a48-9972-eaf53413e548-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.747757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747587 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.747757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747588 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:13.747757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747603 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.747757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.747619 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:13.749309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.749280 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:13.749432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.749369 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "kube-api-access-ql4lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:13.749432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.749375 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:13.749432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.749371 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy" pod "dcffb9cf-bd01-4a48-9972-eaf53413e548" (UID: "dcffb9cf-bd01-4a48-9972-eaf53413e548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:13.849049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849014 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.849049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849041 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.849049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849050 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.849294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849058 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.849294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849067 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.849294 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:13.849075 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4lh\" (UniqueName: \"kubernetes.io/projected/dcffb9cf-bd01-4a48-9972-eaf53413e548-kube-api-access-ql4lh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:14.121662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.121632 5859 generic.go:356] "Generic (PLEG): container finished" podID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerID="d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1" exitCode=0 Nov 02 22:32:14.121849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.121678 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" Nov 02 22:32:14.121849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.121723 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerDied","Data":"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1"} Nov 02 22:32:14.121849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.121787 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z" event={"ID":"dcffb9cf-bd01-4a48-9972-eaf53413e548","Type":"ContainerDied","Data":"61eba1bec86bb6f08347838ef0a46da0f52a30b0c1665d4a510ad34caaad9bad"} Nov 02 22:32:14.121849 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.121817 5859 scope.go:122] "RemoveContainer" containerID="d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1" Nov 02 22:32:14.131145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.131126 5859 scope.go:122] "RemoveContainer" containerID="9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b" Nov 02 22:32:14.140642 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.140625 5859 scope.go:122] "RemoveContainer" containerID="d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1" Nov 02 22:32:14.140894 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:14.140878 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1\": container with ID starting with d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1 not found: ID does not exist" containerID="d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1" Nov 02 22:32:14.140978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.140900 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1"} err="failed to get container status \"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1\": rpc error: code = NotFound desc = could not find container \"d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1\": container with ID starting with d678fbd90b45b71bf668dcaa15c7e857c76127a9b7ef7b6be1e5898277994fc1 not found: ID does not exist" Nov 02 22:32:14.140978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.140919 5859 scope.go:122] "RemoveContainer" containerID="9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b" Nov 02 22:32:14.141193 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:14.141176 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b\": container with ID starting with 9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b not found: ID does not exist" containerID="9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b" Nov 02 22:32:14.141240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.141201 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b"} err="failed to get container status \"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b\": rpc error: code = NotFound desc = could not find container \"9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b\": container with ID starting with 9acbb64bd140443305447d3b3ec2927673c4f86c1420d8eda09a0eb3746bba1b not found: ID does not exist" Nov 02 22:32:14.275165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.275146 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 22:32:14.321634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.321611 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-jfb4z"] Nov 02 22:32:14.531284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531230 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-1e9af3"] Nov 02 22:32:14.531626 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531612 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.531668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531627 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.531668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531642 5859 cpu_manager.go:414] "containerMap: removing container" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.531668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531648 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.531668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531657 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.531668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531662 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531671 5859 cpu_manager.go:414] "containerMap: removing container" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531676 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531683 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531687 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531697 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531701 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531708 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531713 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531724 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531728 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531737 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531742 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531748 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e13a11b4-7cc5-40fc-8183-27323975ce1e" containerName="maas-api" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531752 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13a11b4-7cc5-40fc-8183-27323975ce1e" containerName="maas-api" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531763 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" containerName="maas-api" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531767 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" containerName="maas-api" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531773 5859 cpu_manager.go:414] "containerMap: removing container" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531777 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531785 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531789 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531796 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531800 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531809 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531813 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531822 5859 cpu_manager.go:414] "containerMap: removing container" podUID="9fc7b595-5f5c-48cc-9297-e30e154c6053" containerName="maas-api" Nov 02 22:32:14.531821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531828 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc7b595-5f5c-48cc-9297-e30e154c6053" containerName="maas-api" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531836 5859 cpu_manager.go:414] "containerMap: removing container" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531841 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531846 5859 cpu_manager.go:414] "containerMap: removing container" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531851 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531857 5859 cpu_manager.go:414] "containerMap: removing container" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531862 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531914 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531921 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3eca70f-a55d-40a6-9e01-1504eddb3704" containerName="payload-pre-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531929 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13a11b4-7cc5-40fc-8183-27323975ce1e" containerName="maas-api" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531934 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531941 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff48d4fe-28a8-48d7-9290-18bdd2483ea9" containerName="maas-api" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531945 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e5796c-0890-40f2-91b0-19f8669d0524" containerName="payload-pre-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531964 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531970 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a1c035-2d96-4e95-8da5-a0642d874fb5" containerName="main" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531976 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b295cb13-d8bb-42e7-ac51-c87718503401" containerName="payload-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531982 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc7b595-5f5c-48cc-9297-e30e154c6053" containerName="maas-api" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531988 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531993 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff8d3b8-b702-432d-8965-b5fc4c3072ef" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.531998 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd53e7b-339d-4433-900b-141ea31a8b8a" containerName="payload-pre-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532004 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532010 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="803259bc-9cdf-4d90-affb-b16e1b3b8f0f" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532015 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" containerName="istio-proxy" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532019 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1801c51a-c2ac-441f-a8f0-316c35aec026" containerName="payload-processing" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532025 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.532616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.532030 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8995831-b4f1-4c0d-83bf-98b815f4ad20" containerName="cleanup" Nov 02 22:32:14.536683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.536667 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:14.568882 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.568857 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcffb9cf-bd01-4a48-9972-eaf53413e548" path="/var/lib/kubelet/pods/dcffb9cf-bd01-4a48-9972-eaf53413e548/volumes" Nov 02 22:32:14.586804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.586071 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\"" Nov 02 22:32:14.592585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.592557 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-1e9af3"] Nov 02 22:32:14.655574 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.655544 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n66h\" (UniqueName: \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\") pod \"test-curl-7477-1e9af3\" (UID: \"1081e53b-8816-41ed-8e35-f4e47eaa8d43\") " pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:14.755895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.755870 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n66h\" (UniqueName: \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\") pod \"test-curl-7477-1e9af3\" (UID: \"1081e53b-8816-41ed-8e35-f4e47eaa8d43\") " pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:14.809141 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.809092 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n66h\" (UniqueName: \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\") pod \"test-curl-7477-1e9af3\" (UID: \"1081e53b-8816-41ed-8e35-f4e47eaa8d43\") " pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:14.847943 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:14.847918 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:15.210155 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:15.210128 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-1e9af3"] Nov 02 22:32:15.215652 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:32:15.215627 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1081e53b_8816_41ed_8e35_f4e47eaa8d43.slice/crio-84a5f5707d6644d9f24cd898653ea063524c786f5ddeb0451d9d5142e5b0b73f WatchSource:0}: Error finding container 84a5f5707d6644d9f24cd898653ea063524c786f5ddeb0451d9d5142e5b0b73f: Status 404 returned error can't find the container with id 84a5f5707d6644d9f24cd898653ea063524c786f5ddeb0451d9d5142e5b0b73f Nov 02 22:32:16.134838 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:16.134801 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-1e9af3" event={"ID":"1081e53b-8816-41ed-8e35-f4e47eaa8d43","Type":"ContainerStarted","Data":"6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa"} Nov 02 22:32:16.135335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:16.134849 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-1e9af3" event={"ID":"1081e53b-8816-41ed-8e35-f4e47eaa8d43","Type":"ContainerStarted","Data":"84a5f5707d6644d9f24cd898653ea063524c786f5ddeb0451d9d5142e5b0b73f"} Nov 02 22:32:16.239766 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:16.239714 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-1e9af3" podStartSLOduration=2.239697822 podStartE2EDuration="2.239697822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:16.238633287 +0000 UTC m=+4975.786883067" watchObservedRunningTime="2026-11-02 22:32:16.239697822 +0000 UTC m=+4975.787947605" Nov 02 22:32:17.641907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.641874 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-1e9af3"] Nov 02 22:32:17.646865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.646838 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-1e9af3"] Nov 02 22:32:17.786293 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786266 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-df9ac9"] Nov 02 22:32:17.786676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786663 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:17.786728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786678 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:17.786728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786705 5859 cpu_manager.go:414] "containerMap: removing container" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" containerName="test-curl-7477-1e9af3" Nov 02 22:32:17.786728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786710 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" containerName="test-curl-7477-1e9af3" Nov 02 22:32:17.786839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786784 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2abac4-db3e-4ee9-8e05-485d783ecbbc" containerName="cleanup" Nov 02 22:32:17.786839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.786796 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" containerName="test-curl-7477-1e9af3" Nov 02 22:32:17.790080 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.790060 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:17.841040 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.841017 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-df9ac9"] Nov 02 22:32:17.883862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.883838 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6nc\" (UniqueName: \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\") pod \"test-curl-7477-df9ac9\" (UID: \"302b350e-6cd6-4fb7-bb89-417cb11729cc\") " pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:17.984976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:17.984903 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6nc\" (UniqueName: \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\") pod \"test-curl-7477-df9ac9\" (UID: \"302b350e-6cd6-4fb7-bb89-417cb11729cc\") " pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:18.037486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:18.037465 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6nc\" (UniqueName: \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\") pod \"test-curl-7477-df9ac9\" (UID: \"302b350e-6cd6-4fb7-bb89-417cb11729cc\") " pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:18.101858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:18.101837 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:18.147503 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:18.147471 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-1e9af3" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" containerName="test-curl-7477-1e9af3" containerID="cri-o://6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa" gracePeriod=2 Nov 02 22:32:18.195084 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:18.195041 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-1e9af3\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:18.466123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:18.466096 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-df9ac9"] Nov 02 22:32:18.471894 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:32:18.471867 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod302b350e_6cd6_4fb7_bb89_417cb11729cc.slice/crio-840e16068cb23d187c8120608bdca34dadbeb6d0da9752c0d0872ac799f44d7c WatchSource:0}: Error finding container 840e16068cb23d187c8120608bdca34dadbeb6d0da9752c0d0872ac799f44d7c: Status 404 returned error can't find the container with id 840e16068cb23d187c8120608bdca34dadbeb6d0da9752c0d0872ac799f44d7c Nov 02 22:32:19.154817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:19.154787 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-df9ac9" event={"ID":"302b350e-6cd6-4fb7-bb89-417cb11729cc","Type":"ContainerStarted","Data":"362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485"} Nov 02 22:32:19.154817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:19.154824 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-df9ac9" event={"ID":"302b350e-6cd6-4fb7-bb89-417cb11729cc","Type":"ContainerStarted","Data":"840e16068cb23d187c8120608bdca34dadbeb6d0da9752c0d0872ac799f44d7c"} Nov 02 22:32:19.259232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:19.259190 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7477-df9ac9" podStartSLOduration=2.2591774620000002 podStartE2EDuration="2.259177462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:19.258868555 +0000 UTC m=+4978.807118359" watchObservedRunningTime="2026-11-02 22:32:19.259177462 +0000 UTC m=+4978.807427242" Nov 02 22:32:20.377831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.377804 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:20.507977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.507891 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\" (UniqueName: \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\") pod \"1081e53b-8816-41ed-8e35-f4e47eaa8d43\" (UID: \"1081e53b-8816-41ed-8e35-f4e47eaa8d43\") " Nov 02 22:32:20.509719 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.509693 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h" pod "1081e53b-8816-41ed-8e35-f4e47eaa8d43" (UID: "1081e53b-8816-41ed-8e35-f4e47eaa8d43"). InnerVolumeSpecName "kube-api-access-7n66h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:20.570333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.570294 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" path="/var/lib/kubelet/pods/1081e53b-8816-41ed-8e35-f4e47eaa8d43/volumes" Nov 02 22:32:20.580142 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.580110 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-df9ac9"] Nov 02 22:32:20.584374 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.584350 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-df9ac9"] Nov 02 22:32:20.609472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:20.609446 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n66h\" (UniqueName: \"kubernetes.io/projected/1081e53b-8816-41ed-8e35-f4e47eaa8d43-kube-api-access-7n66h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:21.167284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.167254 5859 generic.go:356] "Generic (PLEG): container finished" podID="1081e53b-8816-41ed-8e35-f4e47eaa8d43" containerID="6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa" exitCode=137 Nov 02 22:32:21.167284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.167292 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-1e9af3" Nov 02 22:32:21.167505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.167357 5859 scope.go:122] "RemoveContainer" containerID="6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa" Nov 02 22:32:21.167505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.167397 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7477-df9ac9" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" containerName="test-curl-7477-df9ac9" containerID="cri-o://362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485" gracePeriod=2 Nov 02 22:32:21.176453 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.176431 5859 scope.go:122] "RemoveContainer" containerID="6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa" Nov 02 22:32:21.179485 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:21.179455 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa\": container with ID starting with 6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa not found: ID does not exist" containerID="6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa" Nov 02 22:32:21.179595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:21.179482 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa"} err="failed to get container status \"6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa\": rpc error: code = NotFound desc = could not find container \"6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa\": container with ID starting with 6b60b21a7f721798d2c4f8b832c6fd489ffbe3beddf1cdec5762c1217742a5fa not found: ID does not exist" Nov 02 22:32:21.215144 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:21.215116 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-df9ac9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:21.262603 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:21.262577 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-df9ac9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:23.379925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:23.379900 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:23.532563 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:23.532495 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\" (UniqueName: \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\") pod \"302b350e-6cd6-4fb7-bb89-417cb11729cc\" (UID: \"302b350e-6cd6-4fb7-bb89-417cb11729cc\") " Nov 02 22:32:23.534413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:23.534389 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc" pod "302b350e-6cd6-4fb7-bb89-417cb11729cc" (UID: "302b350e-6cd6-4fb7-bb89-417cb11729cc"). InnerVolumeSpecName "kube-api-access-sh6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:23.633671 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:23.633648 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6nc\" (UniqueName: \"kubernetes.io/projected/302b350e-6cd6-4fb7-bb89-417cb11729cc-kube-api-access-sh6nc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:24.185062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.185032 5859 generic.go:356] "Generic (PLEG): container finished" podID="302b350e-6cd6-4fb7-bb89-417cb11729cc" containerID="362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485" exitCode=137 Nov 02 22:32:24.185217 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.185071 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:24.185217 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.185102 5859 scope.go:122] "RemoveContainer" containerID="362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485" Nov 02 22:32:24.194000 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.193975 5859 scope.go:122] "RemoveContainer" containerID="362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485" Nov 02 22:32:24.194288 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:24.194266 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485\": container with ID starting with 362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485 not found: ID does not exist" containerID="362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485" Nov 02 22:32:24.194363 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.194296 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485"} err="failed to get container status \"362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485\": rpc error: code = NotFound desc = could not find container \"362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485\": container with ID starting with 362b6174dc26b485c03bca5fa392928fdbf5e827abffcbda47624c069300f485 not found: ID does not exist" Nov 02 22:32:24.241087 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:24.241062 5859 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7477-df9ac9\" is forbidden: User \"system:node:ip-10-0-137-1.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-1.us-west-2.compute.internal' and this object" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" pod="opendatahub/test-curl-7477-df9ac9" Nov 02 22:32:24.568915 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:32:24.568885 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" path="/var/lib/kubelet/pods/302b350e-6cd6-4fb7-bb89-417cb11729cc/volumes" Nov 02 22:32:25.944042 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:25.944006 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:32:55.944222 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:32:55.944183 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:33:06.708705 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:06.708668 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:33:21.727620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:21.727567 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" podUID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" containerName="main" containerID="cri-o://b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c" gracePeriod=45 Nov 02 22:33:21.978469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:21.978413 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:33:22.124585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124557 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.124764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124620 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\" (UniqueName: \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.124764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124659 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.124764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124721 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.124764 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124751 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.125084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124789 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\") pod \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\" (UID: \"c9e8bb29-e531-43d1-b10d-cd8360c8d728\") " Nov 02 22:33:22.125084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.124829 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:22.125084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.125029 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:22.125084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.125044 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:22.125311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.125190 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.125311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.125204 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.125311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.125214 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.126911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.126889 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:22.127064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.126944 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "kube-api-access-g5wwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:22.127064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.127043 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs" pod "c9e8bb29-e531-43d1-b10d-cd8360c8d728" (UID: "c9e8bb29-e531-43d1-b10d-cd8360c8d728"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:22.226060 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.226023 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e8bb29-e531-43d1-b10d-cd8360c8d728-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.226060 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.226053 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wwn\" (UniqueName: \"kubernetes.io/projected/c9e8bb29-e531-43d1-b10d-cd8360c8d728-kube-api-access-g5wwn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.226060 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.226062 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e8bb29-e531-43d1-b10d-cd8360c8d728-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:22.487947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.487917 5859 generic.go:356] "Generic (PLEG): container finished" podID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" containerID="b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c" exitCode=0 Nov 02 22:33:22.488145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.487974 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" Nov 02 22:33:22.488145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.487978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" event={"ID":"c9e8bb29-e531-43d1-b10d-cd8360c8d728","Type":"ContainerDied","Data":"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c"} Nov 02 22:33:22.488145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.488019 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm" event={"ID":"c9e8bb29-e531-43d1-b10d-cd8360c8d728","Type":"ContainerDied","Data":"2a45fa94627761d8a55b3f2de17a3b31532021b677f886066c36a818ef978b5f"} Nov 02 22:33:22.488145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.488037 5859 scope.go:122] "RemoveContainer" containerID="b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c" Nov 02 22:33:22.497736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.497707 5859 scope.go:122] "RemoveContainer" containerID="b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c" Nov 02 22:33:22.498006 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:33:22.497986 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c\": container with ID starting with b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c not found: ID does not exist" containerID="b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c" Nov 02 22:33:22.498072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.498014 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c"} err="failed to get container status \"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c\": rpc error: code = NotFound desc = could not find container \"b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c\": container with ID starting with b616a910d9a3a79bbe00896cc1610fdae67bc538e90586400fc8caa2c4d15a1c not found: ID does not exist" Nov 02 22:33:22.640410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.640377 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:33:22.643337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:22.643314 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-97995b/sub-test-model-97995b-kserve-d997984cf-6dpbm"] Nov 02 22:33:24.569619 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:24.569585 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" path="/var/lib/kubelet/pods/c9e8bb29-e531-43d1-b10d-cd8360c8d728/volumes" Nov 02 22:33:25.943514 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:33:25.943482 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:33:26.168420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:26.168390 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:33:33.080623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:33.080539 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 22:33:38.081406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:38.081347 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" containerID="cri-o://3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060" gracePeriod=25 Nov 02 22:33:41.187318 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.187279 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" podUID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" containerName="main" containerID="cri-o://2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67" gracePeriod=45 Nov 02 22:33:41.434434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.434402 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:33:41.504492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504410 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.504492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504479 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.504741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504500 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\" (UniqueName: \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.504741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504551 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.504741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504588 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.504741 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504642 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\") pod \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\" (UID: \"8fe24530-0cdd-40dd-bef8-774d0c87e8bf\") " Nov 02 22:33:41.505007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504865 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:41.505007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.504937 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.505119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.505089 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:41.505180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.505114 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:41.506853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.506828 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:41.506853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.506846 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "kube-api-access-jg7wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:41.507124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.507109 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs" pod "8fe24530-0cdd-40dd-bef8-774d0c87e8bf" (UID: "8fe24530-0cdd-40dd-bef8-774d0c87e8bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:41.585695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.585664 5859 generic.go:356] "Generic (PLEG): container finished" podID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" containerID="2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67" exitCode=0 Nov 02 22:33:41.585865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.585722 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" event={"ID":"8fe24530-0cdd-40dd-bef8-774d0c87e8bf","Type":"ContainerDied","Data":"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67"} Nov 02 22:33:41.585865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.585738 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" Nov 02 22:33:41.585865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.585750 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg" event={"ID":"8fe24530-0cdd-40dd-bef8-774d0c87e8bf","Type":"ContainerDied","Data":"ca58a4ed510fffaf6f608ed03bfdf705a2706e9443f8915697cdbaacafc7e8ad"} Nov 02 22:33:41.585865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.585765 5859 scope.go:122] "RemoveContainer" containerID="2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67" Nov 02 22:33:41.596836 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.596814 5859 scope.go:122] "RemoveContainer" containerID="2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67" Nov 02 22:33:41.597190 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:33:41.597164 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67\": container with ID starting with 2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67 not found: ID does not exist" containerID="2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67" Nov 02 22:33:41.597250 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.597199 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67"} err="failed to get container status \"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67\": rpc error: code = NotFound desc = could not find container \"2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67\": container with ID starting with 2702f7270b3e5c2ce1bfa6cb327621baf19cebd4dc19cc360191c4f0d4c9ea67 not found: ID does not exist" Nov 02 22:33:41.605451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.605430 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.605451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.605451 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.605586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.605460 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.605586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.605470 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7wj\" (UniqueName: \"kubernetes.io/projected/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-kube-api-access-jg7wj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.605586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.605479 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe24530-0cdd-40dd-bef8-774d0c87e8bf-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:41.741160 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.741129 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:33:41.743838 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:41.743815 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/sub-test-model-a1d751-kserve-bfbc97467-pjrtg"] Nov 02 22:33:42.568430 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:42.568402 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" path="/var/lib/kubelet/pods/8fe24530-0cdd-40dd-bef8-774d0c87e8bf/volumes" Nov 02 22:33:48.000652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:48.000617 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:55.945584 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:33:55.945549 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:33:58.000577 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.000544 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:33:58.328984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.328961 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 22:33:58.459419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.459386 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\") pod \"f65af2a2-6481-4664-92d8-9f95ccc661f8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " Nov 02 22:33:58.459585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.459431 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\" (UniqueName: \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\") pod \"f65af2a2-6481-4664-92d8-9f95ccc661f8\" (UID: \"f65af2a2-6481-4664-92d8-9f95ccc661f8\") " Nov 02 22:33:58.459747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.459725 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume" pod "f65af2a2-6481-4664-92d8-9f95ccc661f8" (UID: "f65af2a2-6481-4664-92d8-9f95ccc661f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:58.461469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.461444 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm" pod "f65af2a2-6481-4664-92d8-9f95ccc661f8" (UID: "f65af2a2-6481-4664-92d8-9f95ccc661f8"). InnerVolumeSpecName "kube-api-access-5nlxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:58.561062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.561003 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65af2a2-6481-4664-92d8-9f95ccc661f8-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:58.561062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.561029 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlxm\" (UniqueName: \"kubernetes.io/projected/f65af2a2-6481-4664-92d8-9f95ccc661f8-kube-api-access-5nlxm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:58.672248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.672213 5859 generic.go:356] "Generic (PLEG): container finished" podID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerID="3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060" exitCode=0 Nov 02 22:33:58.672415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.672285 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" Nov 02 22:33:58.672415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.672292 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" event={"ID":"f65af2a2-6481-4664-92d8-9f95ccc661f8","Type":"ContainerDied","Data":"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060"} Nov 02 22:33:58.672415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.672342 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8" event={"ID":"f65af2a2-6481-4664-92d8-9f95ccc661f8","Type":"ContainerDied","Data":"ee28ad128778cc2f8cc43c6d7cfa4c294130d8eba821e255bdf271f98cac19ff"} Nov 02 22:33:58.672415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.672364 5859 scope.go:122] "RemoveContainer" containerID="3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060" Nov 02 22:33:58.681110 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.681089 5859 scope.go:122] "RemoveContainer" containerID="3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060" Nov 02 22:33:58.681376 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:33:58.681357 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060\": container with ID starting with 3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060 not found: ID does not exist" containerID="3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060" Nov 02 22:33:58.681440 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.681387 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060"} err="failed to get container status \"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060\": rpc error: code = NotFound desc = could not find container \"3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060\": container with ID starting with 3f86f0136edb76730df876c728d0d2715e6a5df07333c36cf074d5e8a6613060 not found: ID does not exist" Nov 02 22:33:58.823644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.823618 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 22:33:58.869884 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:33:58.869859 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-97995b-7cb6597549-pprx8"] Nov 02 22:34:00.568639 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:00.568607 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" path="/var/lib/kubelet/pods/f65af2a2-6481-4664-92d8-9f95ccc661f8/volumes" Nov 02 22:34:01.679666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:01.679631 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 22:34:01.680076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:01.679815 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" podUID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" containerName="maas-api" containerID="cri-o://54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384" gracePeriod=30 Nov 02 22:34:01.926325 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:01.926301 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 22:34:02.093742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.093716 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") pod \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " Nov 02 22:34:02.094009 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.093757 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\" (UniqueName: \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\") pod \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " Nov 02 22:34:02.094009 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.093795 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\") pod \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " Nov 02 22:34:02.094009 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.093909 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\") pod \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\" (UID: \"d2779c71-cef4-4baa-9f78-a96efd64bdc4\") " Nov 02 22:34:02.094227 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.094200 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca" pod "d2779c71-cef4-4baa-9f78-a96efd64bdc4" (UID: "d2779c71-cef4-4baa-9f78-a96efd64bdc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:02.096037 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.096007 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls" pod "d2779c71-cef4-4baa-9f78-a96efd64bdc4" (UID: "d2779c71-cef4-4baa-9f78-a96efd64bdc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:02.096149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.096108 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs" pod "d2779c71-cef4-4baa-9f78-a96efd64bdc4" (UID: "d2779c71-cef4-4baa-9f78-a96efd64bdc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:02.096149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.096138 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6" pod "d2779c71-cef4-4baa-9f78-a96efd64bdc4" (UID: "d2779c71-cef4-4baa-9f78-a96efd64bdc4"). InnerVolumeSpecName "kube-api-access-jp9l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:02.195274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.195251 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:02.195274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.195272 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2779c71-cef4-4baa-9f78-a96efd64bdc4-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:02.195405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.195283 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9l6\" (UniqueName: \"kubernetes.io/projected/d2779c71-cef4-4baa-9f78-a96efd64bdc4-kube-api-access-jp9l6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:02.195405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.195292 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2779c71-cef4-4baa-9f78-a96efd64bdc4-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:02.693657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.693628 5859 generic.go:356] "Generic (PLEG): container finished" podID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" containerID="54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384" exitCode=0 Nov 02 22:34:02.694107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.693674 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" Nov 02 22:34:02.694107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.693703 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" event={"ID":"d2779c71-cef4-4baa-9f78-a96efd64bdc4","Type":"ContainerDied","Data":"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384"} Nov 02 22:34:02.694107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.693754 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6" event={"ID":"d2779c71-cef4-4baa-9f78-a96efd64bdc4","Type":"ContainerDied","Data":"eb93f4c0a676684ab84b53fd73ad47193e72a6b1fd6cb7b8ffea4c2b2dfe9bab"} Nov 02 22:34:02.694107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.693775 5859 scope.go:122] "RemoveContainer" containerID="54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384" Nov 02 22:34:02.703108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.703086 5859 scope.go:122] "RemoveContainer" containerID="54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384" Nov 02 22:34:02.703342 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:34:02.703319 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384\": container with ID starting with 54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384 not found: ID does not exist" containerID="54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384" Nov 02 22:34:02.703415 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.703355 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384"} err="failed to get container status \"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384\": rpc error: code = NotFound desc = could not find container \"54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384\": container with ID starting with 54b16d5620776680e58e2633ebc34765a19201f0c8063997d8d5958126c7c384 not found: ID does not exist" Nov 02 22:34:02.845646 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.845580 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 22:34:02.891862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:02.891829 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a1d751-5d5cf5cc89-cdft6"] Nov 02 22:34:04.569650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:04.569616 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" path="/var/lib/kubelet/pods/d2779c71-cef4-4baa-9f78-a96efd64bdc4/volumes" Nov 02 22:34:06.946907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.946876 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2"] Nov 02 22:34:06.947341 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947306 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" containerName="main" Nov 02 22:34:06.947341 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947318 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" containerName="main" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947345 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947350 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947362 5859 cpu_manager.go:414] "containerMap: removing container" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" containerName="test-curl-7477-df9ac9" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947367 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" containerName="test-curl-7477-df9ac9" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947378 5859 cpu_manager.go:414] "containerMap: removing container" podUID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" containerName="main" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947383 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" containerName="main" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947389 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" containerName="maas-api" Nov 02 22:34:06.947419 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947393 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" containerName="maas-api" Nov 02 22:34:06.947664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947452 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e8bb29-e531-43d1-b10d-cd8360c8d728" containerName="main" Nov 02 22:34:06.947664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947464 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65af2a2-6481-4664-92d8-9f95ccc661f8" containerName="payload-processing" Nov 02 22:34:06.947664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947471 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="302b350e-6cd6-4fb7-bb89-417cb11729cc" containerName="test-curl-7477-df9ac9" Nov 02 22:34:06.947664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947478 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe24530-0cdd-40dd-bef8-774d0c87e8bf" containerName="main" Nov 02 22:34:06.947664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.947484 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2779c71-cef4-4baa-9f78-a96efd64bdc4" containerName="maas-api" Nov 02 22:34:06.950588 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:06.950568 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.002690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.002658 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:34:07.038007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.037980 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.038172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.038025 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.038172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.038136 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gqf\" (UniqueName: \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.048869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.048844 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:34:07.049031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.048976 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:34:07.049087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.049061 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:34:07.050645 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.050624 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2"] Nov 02 22:34:07.139254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.139222 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gqf\" (UniqueName: \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.139426 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.139325 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.139426 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.139370 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.140036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.140015 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.141711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.141688 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.192889 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.192860 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gqf\" (UniqueName: \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\") pod \"maas-api-revoke-sub-aedc7c91-cn7m2\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.263250 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.263174 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:07.626109 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.626070 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2"] Nov 02 22:34:07.632812 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:34:07.632782 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda147d492_a0e4_4c72_8bfe_a708d8c89516.slice/crio-84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d WatchSource:0}: Error finding container 84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d: Status 404 returned error can't find the container with id 84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d Nov 02 22:34:07.719735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:07.719699 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" event={"ID":"a147d492-a0e4-4c72-8bfe-a708d8c89516","Type":"ContainerStarted","Data":"84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d"} Nov 02 22:34:08.726404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:08.726315 5859 generic.go:356] "Generic (PLEG): container finished" podID="a147d492-a0e4-4c72-8bfe-a708d8c89516" containerID="d8c33206445f04e168f67473b98f634957387b49dfb549497d2a22712a88e93c" exitCode=0 Nov 02 22:34:08.726806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:08.726395 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" event={"ID":"a147d492-a0e4-4c72-8bfe-a708d8c89516","Type":"ContainerDied","Data":"d8c33206445f04e168f67473b98f634957387b49dfb549497d2a22712a88e93c"} Nov 02 22:34:09.857110 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.857087 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:09.966714 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.966679 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\" (UniqueName: \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\") pod \"a147d492-a0e4-4c72-8bfe-a708d8c89516\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " Nov 02 22:34:09.966895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.966753 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\") pod \"a147d492-a0e4-4c72-8bfe-a708d8c89516\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " Nov 02 22:34:09.966895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.966823 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\") pod \"a147d492-a0e4-4c72-8bfe-a708d8c89516\" (UID: \"a147d492-a0e4-4c72-8bfe-a708d8c89516\") " Nov 02 22:34:09.967146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.967123 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca" pod "a147d492-a0e4-4c72-8bfe-a708d8c89516" (UID: "a147d492-a0e4-4c72-8bfe-a708d8c89516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:09.968901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.968874 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca" pod "a147d492-a0e4-4c72-8bfe-a708d8c89516" (UID: "a147d492-a0e4-4c72-8bfe-a708d8c89516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:09.969108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:09.969091 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf" pod "a147d492-a0e4-4c72-8bfe-a708d8c89516" (UID: "a147d492-a0e4-4c72-8bfe-a708d8c89516"). InnerVolumeSpecName "kube-api-access-w4gqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:10.067928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.067901 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4gqf\" (UniqueName: \"kubernetes.io/projected/a147d492-a0e4-4c72-8bfe-a708d8c89516-kube-api-access-w4gqf\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:10.067928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.067929 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:10.068149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.067942 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a147d492-a0e4-4c72-8bfe-a708d8c89516-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:10.739365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.739333 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" Nov 02 22:34:10.739544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.739340 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2" event={"ID":"a147d492-a0e4-4c72-8bfe-a708d8c89516","Type":"ContainerDied","Data":"84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d"} Nov 02 22:34:10.739544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:10.739458 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84bbd7b4b15d055bc8d8c30abd72c588416adc90c847d0a3ff43636eeb67179d" Nov 02 22:34:19.261644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.261607 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn"] Nov 02 22:34:19.262053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.262022 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a147d492-a0e4-4c72-8bfe-a708d8c89516" containerName="revoke-keys" Nov 02 22:34:19.262053 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.262033 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a147d492-a0e4-4c72-8bfe-a708d8c89516" containerName="revoke-keys" Nov 02 22:34:19.262137 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.262109 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a147d492-a0e4-4c72-8bfe-a708d8c89516" containerName="revoke-keys" Nov 02 22:34:19.265300 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.265282 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.316084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.316052 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:34:19.318305 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.318282 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn"] Nov 02 22:34:19.356892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.356867 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.357043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.356905 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.357095 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.357064 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q7n\" (UniqueName: \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.362666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.362646 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:34:19.362786 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.362678 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:34:19.362786 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.362686 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:34:19.458111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.458077 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q7n\" (UniqueName: \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.458280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.458190 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.458280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.458233 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.458801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.458780 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.460533 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.460512 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.510894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.510864 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q7n\" (UniqueName: \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\") pod \"maas-api-revoke-sub-afa584d3-548tn\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.577320 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.577299 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:19.944591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:19.944507 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn"] Nov 02 22:34:19.950750 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:34:19.950723 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f2f1ab_bf51_4a37_a4fd_da07fed19f3d.slice/crio-ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c WatchSource:0}: Error finding container ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c: Status 404 returned error can't find the container with id ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c Nov 02 22:34:20.373842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:20.373809 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 22:34:20.790490 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:20.790460 5859 generic.go:356] "Generic (PLEG): container finished" podID="60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" containerID="a79b3274acea0c9a9d1ae0fadce405d0413b3bac4f14e27f9c557d5ec5b337ce" exitCode=0 Nov 02 22:34:20.790655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:20.790546 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" event={"ID":"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d","Type":"ContainerDied","Data":"a79b3274acea0c9a9d1ae0fadce405d0413b3bac4f14e27f9c557d5ec5b337ce"} Nov 02 22:34:20.790655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:20.790638 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" event={"ID":"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d","Type":"ContainerStarted","Data":"ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c"} Nov 02 22:34:21.921145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.921118 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:21.983249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.983223 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\" (UniqueName: \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\") pod \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " Nov 02 22:34:21.983388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.983283 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\") pod \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " Nov 02 22:34:21.983388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.983306 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\") pod \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\" (UID: \"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d\") " Nov 02 22:34:21.983698 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.983670 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca" pod "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" (UID: "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:21.985335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.985315 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n" pod "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" (UID: "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d"). InnerVolumeSpecName "kube-api-access-c2q7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:21.985439 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:21.985341 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca" pod "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" (UID: "60f2f1ab-bf51-4a37-a4fd-da07fed19f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:22.084487 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.084458 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:22.084487 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.084486 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:22.084697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.084498 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2q7n\" (UniqueName: \"kubernetes.io/projected/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d-kube-api-access-c2q7n\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:22.801664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.801586 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" Nov 02 22:34:22.801664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.801592 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn" event={"ID":"60f2f1ab-bf51-4a37-a4fd-da07fed19f3d","Type":"ContainerDied","Data":"ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c"} Nov 02 22:34:22.801664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:22.801634 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5686463c6288b90971131eac6cd0904a555d90276a9454c3f6a17d33c2dc3c" Nov 02 22:34:25.374648 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:25.374584 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" containerID="cri-o://4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0" gracePeriod=25 Nov 02 22:34:25.943833 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:34:25.943795 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:34:33.145832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:33.145789 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:34:43.145690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:43.145650 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:34:45.608396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.608368 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 22:34:45.704058 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.703997 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\" (UniqueName: \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\") pod \"abf47427-58b4-4e55-8258-b47a87a21013\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " Nov 02 22:34:45.704180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.704139 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\") pod \"abf47427-58b4-4e55-8258-b47a87a21013\" (UID: \"abf47427-58b4-4e55-8258-b47a87a21013\") " Nov 02 22:34:45.704462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.704443 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume" pod "abf47427-58b4-4e55-8258-b47a87a21013" (UID: "abf47427-58b4-4e55-8258-b47a87a21013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:45.706003 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.705982 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb" pod "abf47427-58b4-4e55-8258-b47a87a21013" (UID: "abf47427-58b4-4e55-8258-b47a87a21013"). InnerVolumeSpecName "kube-api-access-7rwrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:45.805635 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.805610 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwrb\" (UniqueName: \"kubernetes.io/projected/abf47427-58b4-4e55-8258-b47a87a21013-kube-api-access-7rwrb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.805635 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.805633 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf47427-58b4-4e55-8258-b47a87a21013-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.923672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.923645 5859 generic.go:356] "Generic (PLEG): container finished" podID="abf47427-58b4-4e55-8258-b47a87a21013" containerID="4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0" exitCode=0 Nov 02 22:34:45.923787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.923690 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" Nov 02 22:34:45.923787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.923725 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" event={"ID":"abf47427-58b4-4e55-8258-b47a87a21013","Type":"ContainerDied","Data":"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0"} Nov 02 22:34:45.923787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.923777 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv" event={"ID":"abf47427-58b4-4e55-8258-b47a87a21013","Type":"ContainerDied","Data":"94f424f01c8a786b9e7d65fff233a9f398aa76b76dbd9ca1887ff23821fc74c7"} Nov 02 22:34:45.923910 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.923798 5859 scope.go:122] "RemoveContainer" containerID="4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0" Nov 02 22:34:45.932544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.932523 5859 scope.go:122] "RemoveContainer" containerID="4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0" Nov 02 22:34:45.932822 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:34:45.932802 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0\": container with ID starting with 4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0 not found: ID does not exist" containerID="4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0" Nov 02 22:34:45.932882 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:45.932831 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0"} err="failed to get container status \"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0\": rpc error: code = NotFound desc = could not find container \"4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0\": container with ID starting with 4aedfb4c9e9015ae5218265dd1108dfb1373eab897801dd54c4bfba931b783d0 not found: ID does not exist" Nov 02 22:34:46.077748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:46.077724 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 22:34:46.080411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:46.080388 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f51151-548c54cd95-lxtwv"] Nov 02 22:34:46.568862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:46.568838 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf47427-58b4-4e55-8258-b47a87a21013" path="/var/lib/kubelet/pods/abf47427-58b4-4e55-8258-b47a87a21013/volumes" Nov 02 22:34:48.475108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475075 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:34:48.475645 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475617 5859 cpu_manager.go:414] "containerMap: removing container" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" Nov 02 22:34:48.475645 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475643 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" Nov 02 22:34:48.475821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475665 5859 cpu_manager.go:414] "containerMap: removing container" podUID="60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" containerName="revoke-keys" Nov 02 22:34:48.475821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475671 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" containerName="revoke-keys" Nov 02 22:34:48.475821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475749 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf47427-58b4-4e55-8258-b47a87a21013" containerName="payload-pre-processing" Nov 02 22:34:48.475821 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.475760 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" containerName="revoke-keys" Nov 02 22:34:48.480339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.480323 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.529677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.529657 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\"" Nov 02 22:34:48.529677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.529663 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:34:48.578226 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.578201 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:34:48.628731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.628707 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.628853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.628743 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.628853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.628765 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.628937 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.628875 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.629028 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.629007 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.629067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.629040 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.629067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.629057 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.629155 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.629132 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkw97\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.629191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.629168 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729572 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729607 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729629 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729655 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729709 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729726 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729742 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729780 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkw97\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.729881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.729807 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.730210 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.730083 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.730210 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.730178 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.730295 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.730225 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.730353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.730331 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.730432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.730414 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.732206 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.732182 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.732365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.732350 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.783516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.783490 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.783629 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.783584 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkw97\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:48.792390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:48.792365 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:49.211308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.211277 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:34:49.255324 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:34:49.255298 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0410b024_3b42_40b6_b6bc_887f54532636.slice/crio-4d0fabc4f2d6acb04b9e24d1aa51e3f38b5fd0d51a545ef27f8a4ffbf83c0adc WatchSource:0}: Error finding container 4d0fabc4f2d6acb04b9e24d1aa51e3f38b5fd0d51a545ef27f8a4ffbf83c0adc: Status 404 returned error can't find the container with id 4d0fabc4f2d6acb04b9e24d1aa51e3f38b5fd0d51a545ef27f8a4ffbf83c0adc Nov 02 22:34:49.257359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.257328 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:49.257464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.257409 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:49.257464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.257436 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:34:49.947526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.947493 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" event={"ID":"0410b024-3b42-40b6-b6bc-887f54532636","Type":"ContainerStarted","Data":"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b"} Nov 02 22:34:49.948049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:49.947547 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" event={"ID":"0410b024-3b42-40b6-b6bc-887f54532636","Type":"ContainerStarted","Data":"4d0fabc4f2d6acb04b9e24d1aa51e3f38b5fd0d51a545ef27f8a4ffbf83c0adc"} Nov 02 22:34:50.104913 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:50.104860 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" podStartSLOduration=2.104846101 podStartE2EDuration="2.104846101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:34:50.103776206 +0000 UTC m=+5129.652025987" watchObservedRunningTime="2026-11-02 22:34:50.104846101 +0000 UTC m=+5129.653095881" Nov 02 22:34:50.793291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:50.793256 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:34:50.794629 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:50.794607 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 22:34:50.794691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:50.794653 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 22:34:51.732169 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:51.732125 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:34:51.793583 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:51.793552 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 22:34:51.793747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:51.793601 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 22:34:51.957840 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:51.957806 5859 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" secret="" err="secret \"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\" not found" Nov 02 22:34:52.792668 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:52.792632 5859 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 02 22:34:52.793106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:52.792682 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 02 22:34:52.961858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:52.961818 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" containerID="cri-o://8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b" gracePeriod=30 Nov 02 22:34:53.591484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:53.591456 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vsgm4_33a98961-7c4a-4fdd-a2d9-b1072164286d/manager/0.log" Nov 02 22:34:53.826974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:53.825343 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qr5f4_28db1cfd-ba37-4772-91fb-ad9f035e3624/kuadrant-console-plugin/0.log" Nov 02 22:34:55.949822 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:34:55.949784 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:34:58.263879 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:34:58.263850 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mqd69_5a272ed8-590a-4bc5-aace-1f092f17eb9d/manager/0.log" Nov 02 22:35:00.507780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.507758 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:35:00.539182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539155 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539204 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539234 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539308 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539260 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539348 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539381 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539428 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539515 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539525 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:00.539628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539555 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:00.539628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539560 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\") pod \"0410b024-3b42-40b6-b6bc-887f54532636\" (UID: \"0410b024-3b42-40b6-b6bc-887f54532636\") " Nov 02 22:35:00.539855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539621 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:00.539855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539603 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:00.539855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539775 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:00.540126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.539985 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.540126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.540008 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.540126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.540022 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.540126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.540037 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.540126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.540050 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410b024-3b42-40b6-b6bc-887f54532636-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.541400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.541380 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:00.541503 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.541484 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "kube-api-access-gkw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:00.541572 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.541546 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:35:00.541640 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.541627 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy" pod "0410b024-3b42-40b6-b6bc-887f54532636" (UID: "0410b024-3b42-40b6-b6bc-887f54532636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:00.640531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.640488 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410b024-3b42-40b6-b6bc-887f54532636-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.640531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.640507 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410b024-3b42-40b6-b6bc-887f54532636-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.640531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.640516 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:00.640531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:00.640526 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkw97\" (UniqueName: \"kubernetes.io/projected/0410b024-3b42-40b6-b6bc-887f54532636-kube-api-access-gkw97\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:01.000545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.000475 5859 generic.go:356] "Generic (PLEG): container finished" podID="0410b024-3b42-40b6-b6bc-887f54532636" containerID="8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b" exitCode=0 Nov 02 22:35:01.000545 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.000521 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" Nov 02 22:35:01.000700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.000553 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" event={"ID":"0410b024-3b42-40b6-b6bc-887f54532636","Type":"ContainerDied","Data":"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b"} Nov 02 22:35:01.000700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.000593 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v" event={"ID":"0410b024-3b42-40b6-b6bc-887f54532636","Type":"ContainerDied","Data":"4d0fabc4f2d6acb04b9e24d1aa51e3f38b5fd0d51a545ef27f8a4ffbf83c0adc"} Nov 02 22:35:01.000700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.000617 5859 scope.go:122] "RemoveContainer" containerID="8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b" Nov 02 22:35:01.009601 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.009583 5859 scope.go:122] "RemoveContainer" containerID="8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b" Nov 02 22:35:01.009866 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:35:01.009846 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b\": container with ID starting with 8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b not found: ID does not exist" containerID="8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b" Nov 02 22:35:01.009934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.009880 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b"} err="failed to get container status \"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b\": rpc error: code = NotFound desc = could not find container \"8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b\": container with ID starting with 8b0ac04bb24a54f41bb9ccf68e5a782e2626f132a547ac955b8814325c67327b not found: ID does not exist" Nov 02 22:35:01.152319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.152286 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:35:01.156023 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:01.156003 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-6bv7v"] Nov 02 22:35:02.568464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:02.568373 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0410b024-3b42-40b6-b6bc-887f54532636" path="/var/lib/kubelet/pods/0410b024-3b42-40b6-b6bc-887f54532636/volumes" Nov 02 22:35:03.591631 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.591594 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:35:03.592038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.592018 5859 cpu_manager.go:414] "containerMap: removing container" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" Nov 02 22:35:03.592038 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.592030 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" Nov 02 22:35:03.592154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.592142 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="0410b024-3b42-40b6-b6bc-887f54532636" containerName="istio-proxy" Nov 02 22:35:03.596685 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.596667 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.645675 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.645640 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:35:03.670551 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670527 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670561 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670581 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670604 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670715 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brld\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670936 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670736 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.670936 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670809 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.671082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.670931 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.671082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.671021 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.692222 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.692197 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d64a7c7d-openshift-default-dockercfg-7754x\"" Nov 02 22:35:03.694001 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.693981 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:35:03.772152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772127 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772160 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772178 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772196 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772220 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772256 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brld\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772279 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772318 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772394 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772694 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772580 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772694 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772623 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772694 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772672 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.772804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.772703 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.773231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.773208 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.774499 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.774476 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.774755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.774737 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.828485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.828461 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.828600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.828585 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brld\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\") pod \"e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:03.908909 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:03.908859 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:04.329413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:04.329383 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:35:04.335187 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:35:04.335158 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92da629_a764_453b_8201_437f8c0c0077.slice/crio-53a7682ec601ec14e8051dee74d4d36c1c51dd7af15de927d8bd3cbce1a7bfb4 WatchSource:0}: Error finding container 53a7682ec601ec14e8051dee74d4d36c1c51dd7af15de927d8bd3cbce1a7bfb4: Status 404 returned error can't find the container with id 53a7682ec601ec14e8051dee74d4d36c1c51dd7af15de927d8bd3cbce1a7bfb4 Nov 02 22:35:04.337143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:04.337114 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:04.337224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:04.337175 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:04.337224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:04.337203 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:05.023699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:05.023665 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerStarted","Data":"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c"} Nov 02 22:35:05.024180 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:05.023720 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerStarted","Data":"53a7682ec601ec14e8051dee74d4d36c1c51dd7af15de927d8bd3cbce1a7bfb4"} Nov 02 22:35:05.132592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:05.132553 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podStartSLOduration=2.132540814 podStartE2EDuration="2.132540814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:05.13199081 +0000 UTC m=+5144.680240612" watchObservedRunningTime="2026-11-02 22:35:05.132540814 +0000 UTC m=+5144.680790594" Nov 02 22:35:05.909684 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:05.909652 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:05.914212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:05.914188 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:06.028274 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:06.028251 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:06.028921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:06.028908 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:10.920994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:10.920946 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:35:10.924446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:10.924426 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:10.977656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:10.977630 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:35:11.021970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.021940 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d64a7c7d\"" Nov 02 22:35:11.022097 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.022055 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d64a7c7d-dockercfg-2x9hn\"" Nov 02 22:35:11.035476 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.035452 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.035575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.035510 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8ht\" (UniqueName: \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.136327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.136301 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.136445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.136387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8ht\" (UniqueName: \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.136996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.136974 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.189505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.189460 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8ht\" (UniqueName: \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\") pod \"payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.238842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.238824 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:11.605724 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:11.605694 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:35:11.612435 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:35:11.612409 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8fe350_cba9_47b4_8c27_fcfd89c8a563.slice/crio-e364f838545eecbd34157cd4ddfdea617e217a1d0fe1c4a28ba7a86b6a5826bd WatchSource:0}: Error finding container e364f838545eecbd34157cd4ddfdea617e217a1d0fe1c4a28ba7a86b6a5826bd: Status 404 returned error can't find the container with id e364f838545eecbd34157cd4ddfdea617e217a1d0fe1c4a28ba7a86b6a5826bd Nov 02 22:35:12.058680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:12.058602 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" event={"ID":"5a8fe350-cba9-47b4-8c27-fcfd89c8a563","Type":"ContainerStarted","Data":"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7"} Nov 02 22:35:12.058680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:12.058650 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" event={"ID":"5a8fe350-cba9-47b4-8c27-fcfd89c8a563","Type":"ContainerStarted","Data":"e364f838545eecbd34157cd4ddfdea617e217a1d0fe1c4a28ba7a86b6a5826bd"} Nov 02 22:35:12.059094 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:12.058750 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:12.166164 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:12.166106 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" podStartSLOduration=2.166094175 podStartE2EDuration="2.166094175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:12.163418959 +0000 UTC m=+5151.711668750" watchObservedRunningTime="2026-11-02 22:35:12.166094175 +0000 UTC m=+5151.714343955" Nov 02 22:35:13.588564 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.588529 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:13.592194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.592172 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.641204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.641179 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d64a7c7d-serving-cert\"" Nov 02 22:35:13.641318 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.641207 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:35:13.660679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.660655 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4cm\" (UniqueName: \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.660816 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.660715 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.660939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.660925 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.661036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.661004 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.687767 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.687747 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:35:13.687890 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.687840 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:35:13.687941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.687890 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 22:35:13.689735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.689717 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:13.762371 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.762348 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.762517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.762394 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.762517 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:35:13.762473 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-serving-cert: secret "maas-api-e2e-ait-d64a7c7d-serving-cert" not found Nov 02 22:35:13.762517 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:35:13.762514 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls podName:a542c8d1-9a3d-4e89-8da9-acd0ec4159b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:35:14.262500254 +0000 UTC m=+5153.810750014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls") pod "maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" (UID: "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9") : secret "maas-api-e2e-ait-d64a7c7d-serving-cert" not found Nov 02 22:35:13.762713 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.762558 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.762713 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.762650 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4cm\" (UniqueName: \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.763249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.763231 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.764728 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.764708 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:13.816491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:13.816469 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4cm\" (UniqueName: \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:14.267354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:14.267320 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:14.269745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:14.269722 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") pod \"maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:14.504543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:14.504512 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:14.869685 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:14.869654 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:14.875092 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:35:14.875059 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda542c8d1_9a3d_4e89_8da9_acd0ec4159b9.slice/crio-aeea25bab5d1a7b5b8395c215179e5547088e74d5f39fc0cbe6d936e1cd08385 WatchSource:0}: Error finding container aeea25bab5d1a7b5b8395c215179e5547088e74d5f39fc0cbe6d936e1cd08385: Status 404 returned error can't find the container with id aeea25bab5d1a7b5b8395c215179e5547088e74d5f39fc0cbe6d936e1cd08385 Nov 02 22:35:14.876458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:14.876440 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:35:15.074063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:15.074038 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" event={"ID":"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9","Type":"ContainerStarted","Data":"aeea25bab5d1a7b5b8395c215179e5547088e74d5f39fc0cbe6d936e1cd08385"} Nov 02 22:35:16.081100 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:16.081059 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" event={"ID":"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9","Type":"ContainerStarted","Data":"c97151231133ba32db811617157b736a4ec2e96c687ba5fb8cc747476d639abb"} Nov 02 22:35:16.081543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:16.081189 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:16.188752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:16.188703 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" podStartSLOduration=2.233796109 podStartE2EDuration="3.188686031s" totalImagesPullingTime="954.889922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:16.187081432 +0000 UTC m=+5155.735331214" watchObservedRunningTime="2026-11-02 22:35:16.188686031 +0000 UTC m=+5155.736935867" Nov 02 22:35:22.091559 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:22.091526 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:23.066745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:23.066720 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:25.952869 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:35:25.952830 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:35:31.176146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.176119 5859 generic.go:356] "Generic (PLEG): container finished" podID="f92da629-a764-453b-8201-437f8c0c0077" containerID="445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c" exitCode=0 Nov 02 22:35:31.176580 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.176193 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerDied","Data":"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c"} Nov 02 22:35:31.176580 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.176494 5859 scope.go:122] "RemoveContainer" containerID="445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c" Nov 02 22:35:31.177591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.177567 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:31.177667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.177631 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:31.177667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:31.177664 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:35:32.190787 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:32.190746 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerStarted","Data":"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f"} Nov 02 22:35:32.922972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:32.920231 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:32.930065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:32.930037 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:35:32.930194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:32.930097 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:35:33.923975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:33.921021 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:35:33.923975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:33.921090 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:35:34.930352 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:34.930314 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:35:34.930886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:34.930384 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:35:35.925759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:35.925324 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:35:35.925759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:35.925383 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:35:36.029970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:36.029209 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:36.909365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:36.909327 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:35:36.909553 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:36.909393 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:35:37.913505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:37.913476 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:37.914721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:37.914699 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:35:54.153634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.153599 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:54.154105 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.153814 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" podUID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" containerName="maas-api" containerID="cri-o://c97151231133ba32db811617157b736a4ec2e96c687ba5fb8cc747476d639abb" gracePeriod=30 Nov 02 22:35:54.329251 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.329221 5859 generic.go:356] "Generic (PLEG): container finished" podID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" containerID="c97151231133ba32db811617157b736a4ec2e96c687ba5fb8cc747476d639abb" exitCode=0 Nov 02 22:35:54.329406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.329297 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" event={"ID":"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9","Type":"ContainerDied","Data":"c97151231133ba32db811617157b736a4ec2e96c687ba5fb8cc747476d639abb"} Nov 02 22:35:54.398931 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.398907 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:54.440907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.440856 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\") pod \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " Nov 02 22:35:54.441035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.440934 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\" (UniqueName: \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\") pod \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " Nov 02 22:35:54.441035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.441007 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") pod \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " Nov 02 22:35:54.441109 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.441045 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\") pod \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\" (UID: \"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9\") " Nov 02 22:35:54.441442 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.441420 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca" pod "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" (UID: "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:54.442935 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.442912 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs" pod "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" (UID: "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:54.443058 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.442946 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm" pod "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" (UID: "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9"). InnerVolumeSpecName "kube-api-access-9g4cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:54.443108 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.443084 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls" pod "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" (UID: "a542c8d1-9a3d-4e89-8da9-acd0ec4159b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:54.542126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.542093 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:54.542126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.542120 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:54.542126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.542129 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:54.542363 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.542137 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4cm\" (UniqueName: \"kubernetes.io/projected/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9-kube-api-access-9g4cm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:54.610459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:54.610428 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:35:55.335364 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:55.335334 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" Nov 02 22:35:55.335824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:55.335365 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx" event={"ID":"a542c8d1-9a3d-4e89-8da9-acd0ec4159b9","Type":"ContainerDied","Data":"aeea25bab5d1a7b5b8395c215179e5547088e74d5f39fc0cbe6d936e1cd08385"} Nov 02 22:35:55.335824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:55.335427 5859 scope.go:122] "RemoveContainer" containerID="c97151231133ba32db811617157b736a4ec2e96c687ba5fb8cc747476d639abb" Nov 02 22:35:55.487863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:55.487832 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:55.534550 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:55.534516 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d64a7c7d-77f86b594c-bhdgx"] Nov 02 22:35:55.943656 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:35:55.943619 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:35:56.569683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:56.569652 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" path="/var/lib/kubelet/pods/a542c8d1-9a3d-4e89-8da9-acd0ec4159b9/volumes" Nov 02 22:35:59.611181 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.611119 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" podUID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" containerName="payload-processing" containerID="cri-o://15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7" gracePeriod=25 Nov 02 22:35:59.880492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.880467 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:35:59.997153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.997119 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\" (UniqueName: \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\") pod \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " Nov 02 22:35:59.997344 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.997225 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\") pod \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\" (UID: \"5a8fe350-cba9-47b4-8c27-fcfd89c8a563\") " Nov 02 22:35:59.997585 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.997557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume" pod "5a8fe350-cba9-47b4-8c27-fcfd89c8a563" (UID: "5a8fe350-cba9-47b4-8c27-fcfd89c8a563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:59.999673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:35:59.999644 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht" pod "5a8fe350-cba9-47b4-8c27-fcfd89c8a563" (UID: "5a8fe350-cba9-47b4-8c27-fcfd89c8a563"). InnerVolumeSpecName "kube-api-access-ph8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:00.098366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.098333 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8ht\" (UniqueName: \"kubernetes.io/projected/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-kube-api-access-ph8ht\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:00.098366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.098361 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a8fe350-cba9-47b4-8c27-fcfd89c8a563-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:00.363592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.363555 5859 generic.go:356] "Generic (PLEG): container finished" podID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" containerID="15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7" exitCode=0 Nov 02 22:36:00.363592 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.363595 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" event={"ID":"5a8fe350-cba9-47b4-8c27-fcfd89c8a563","Type":"ContainerDied","Data":"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7"} Nov 02 22:36:00.363856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.363602 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" Nov 02 22:36:00.363856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.363674 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp" event={"ID":"5a8fe350-cba9-47b4-8c27-fcfd89c8a563","Type":"ContainerDied","Data":"e364f838545eecbd34157cd4ddfdea617e217a1d0fe1c4a28ba7a86b6a5826bd"} Nov 02 22:36:00.363856 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.363703 5859 scope.go:122] "RemoveContainer" containerID="15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7" Nov 02 22:36:00.372878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.372859 5859 scope.go:122] "RemoveContainer" containerID="15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7" Nov 02 22:36:00.373173 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:00.373152 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7\": container with ID starting with 15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7 not found: ID does not exist" containerID="15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7" Nov 02 22:36:00.373252 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.373183 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7"} err="failed to get container status \"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7\": rpc error: code = NotFound desc = could not find container \"15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7\": container with ID starting with 15658bc7203c75fe5acf7f7502129014bb8d5229012b8613840a3dd54718a6e7 not found: ID does not exist" Nov 02 22:36:00.518625 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.518594 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:36:00.570322 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:00.570296 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d64a7c7d-f5757d8cf-nr2pp"] Nov 02 22:36:02.569755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:02.569723 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" path="/var/lib/kubelet/pods/5a8fe350-cba9-47b4-8c27-fcfd89c8a563/volumes" Nov 02 22:36:04.480550 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:04.480513 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:36:04.481025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:04.480752 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" containerID="cri-o://205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f" gracePeriod=30 Nov 02 22:36:07.916707 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:07.916663 5859 patch_prober.go:29] interesting pod/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:36:07.917128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:07.916731 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:36:09.744423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.744356 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:36:09.897247 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897216 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897326 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897363 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897391 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897438 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897468 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897505 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897646 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.897863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897732 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:09.897863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897765 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:09.898027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897863 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:09.898027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.897925 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:09.898027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898013 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\") pod \"f92da629-a764-453b-8201-437f8c0c0077\" (UID: \"f92da629-a764-453b-8201-437f8c0c0077\") " Nov 02 22:36:09.898179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898043 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:09.898360 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898343 5859 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-credential-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.898428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898364 5859 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f92da629-a764-453b-8201-437f8c0c0077-istiod-ca-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.898428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898379 5859 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.898428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898393 5859 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-data\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.898428 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.898406 5859 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-workload-socket\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.899838 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.899809 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:36:09.900237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.900205 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:36:09.900375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.900241 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:09.900375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.900318 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld" pod "f92da629-a764-453b-8201-437f8c0c0077" (UID: "f92da629-a764-453b-8201-437f8c0c0077"). InnerVolumeSpecName "kube-api-access-7brld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:09.999147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.999061 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7brld\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-kube-api-access-7brld\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.999147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.999095 5859 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f92da629-a764-453b-8201-437f8c0c0077-istio-podinfo\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.999147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.999107 5859 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f92da629-a764-453b-8201-437f8c0c0077-istio-envoy\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.999147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:09.999115 5859 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f92da629-a764-453b-8201-437f8c0c0077-istio-token\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:10.417054 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.417019 5859 generic.go:356] "Generic (PLEG): container finished" podID="f92da629-a764-453b-8201-437f8c0c0077" containerID="205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f" exitCode=0 Nov 02 22:36:10.417236 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.417069 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" Nov 02 22:36:10.417236 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.417108 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerDied","Data":"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f"} Nov 02 22:36:10.417236 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.417169 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl" event={"ID":"f92da629-a764-453b-8201-437f8c0c0077","Type":"ContainerDied","Data":"53a7682ec601ec14e8051dee74d4d36c1c51dd7af15de927d8bd3cbce1a7bfb4"} Nov 02 22:36:10.417236 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.417197 5859 scope.go:122] "RemoveContainer" containerID="205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f" Nov 02 22:36:10.426784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.426767 5859 scope.go:122] "RemoveContainer" containerID="445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c" Nov 02 22:36:10.436035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.436017 5859 scope.go:122] "RemoveContainer" containerID="205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f" Nov 02 22:36:10.436347 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:10.436328 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f\": container with ID starting with 205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f not found: ID does not exist" containerID="205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f" Nov 02 22:36:10.436410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.436357 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f"} err="failed to get container status \"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f\": rpc error: code = NotFound desc = could not find container \"205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f\": container with ID starting with 205667c58d7388ce404513ca64496b49d8678eeedfaefb1698902b12cbb7173f not found: ID does not exist" Nov 02 22:36:10.436410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.436375 5859 scope.go:122] "RemoveContainer" containerID="445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c" Nov 02 22:36:10.436622 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:10.436602 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c\": container with ID starting with 445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c not found: ID does not exist" containerID="445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c" Nov 02 22:36:10.436663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.436628 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c"} err="failed to get container status \"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c\": rpc error: code = NotFound desc = could not find container \"445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c\": container with ID starting with 445e8118b75585bf97379e8aa482803c41d1af8c77454ca6cec7b0c66a833d0c not found: ID does not exist" Nov 02 22:36:10.617848 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.617816 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:36:10.664452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.664413 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d64a7c7d-openshift-default-74969d6fc7-6j6nl"] Nov 02 22:36:10.672112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672028 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:10.672460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672440 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672457 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672491 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672500 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672509 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" containerName="maas-api" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672514 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" containerName="maas-api" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672528 5859 cpu_manager.go:414] "containerMap: removing container" podUID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" containerName="payload-processing" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672533 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" containerName="payload-processing" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672617 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672629 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92da629-a764-453b-8201-437f8c0c0077" containerName="istio-proxy" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672635 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a542c8d1-9a3d-4e89-8da9-acd0ec4159b9" containerName="maas-api" Nov 02 22:36:10.672663 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.672644 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8fe350-cba9-47b4-8c27-fcfd89c8a563" containerName="payload-processing" Nov 02 22:36:10.677296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.677279 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.728410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.728380 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:36:10.728662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.728394 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:36:10.728791 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.728473 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvsgp\"" Nov 02 22:36:10.728791 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.728596 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:36:10.731005 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.730977 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:10.806806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.806770 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.806806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.806813 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgb9h\" (UniqueName: \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.807286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.806893 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.908087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.908051 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.908289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.908197 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.908289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.908242 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgb9h\" (UniqueName: \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.910683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.910654 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.910813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.910691 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.961947 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.961879 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgb9h\" (UniqueName: \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\") pod \"maas-controller-7d468bc7cc-zfcwf\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:10.990265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:10.990235 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:11.359795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:11.359760 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:11.365685 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:36:11.365662 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74e712b_1a7d_4142_8cb2_c7a20ff3e3b8.slice/crio-2b619358d715ab2bdaebdc88122f46b6bb9fc84dddad76507de070f213fe87d0 WatchSource:0}: Error finding container 2b619358d715ab2bdaebdc88122f46b6bb9fc84dddad76507de070f213fe87d0: Status 404 returned error can't find the container with id 2b619358d715ab2bdaebdc88122f46b6bb9fc84dddad76507de070f213fe87d0 Nov 02 22:36:11.423459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:11.423428 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" event={"ID":"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8","Type":"ContainerStarted","Data":"2b619358d715ab2bdaebdc88122f46b6bb9fc84dddad76507de070f213fe87d0"} Nov 02 22:36:12.569633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:12.569603 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92da629-a764-453b-8201-437f8c0c0077" path="/var/lib/kubelet/pods/f92da629-a764-453b-8201-437f8c0c0077/volumes" Nov 02 22:36:18.464101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:18.464058 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" event={"ID":"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8","Type":"ContainerStarted","Data":"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f"} Nov 02 22:36:18.464515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:18.464204 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:18.572831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:18.572778 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" podStartSLOduration=2.372520585 podStartE2EDuration="8.57276458s" totalImagesPullingTime="6.200243995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:18.57158834 +0000 UTC m=+5218.119838131" watchObservedRunningTime="2026-11-02 22:36:18.57276458 +0000 UTC m=+5218.121014361" Nov 02 22:36:25.944976 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:25.944920 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:36:27.530033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.529995 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m"] Nov 02 22:36:27.535391 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.535369 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.577007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.576973 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx"] Nov 02 22:36:27.580773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.580756 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.590254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.590226 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\"" Nov 02 22:36:27.590591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.590575 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:36:27.631198 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.631161 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m"] Nov 02 22:36:27.660268 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660223 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.660470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660292 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.660470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660364 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.660470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660417 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqsv\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-kube-api-access-gjqsv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.660658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660506 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.660658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660650 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drt7m\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-kube-api-access-drt7m\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.660781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660679 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.660781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660712 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.660869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660809 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.660869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660837 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.660869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660852 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.661008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660911 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fff690-40c3-413e-98d5-a4955dda45b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.661008 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.660963 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.661113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.661009 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0914efbd-e8c2-4055-922b-760acb575c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.661165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.661143 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.661231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.661177 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.661231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.661223 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0914efbd-e8c2-4055-922b-760acb575c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.661312 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.661253 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.691757 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.691711 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx"] Nov 02 22:36:27.762275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762235 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762287 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762320 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762485 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762356 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqsv\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-kube-api-access-gjqsv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762484 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762544 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drt7m\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-kube-api-access-drt7m\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762580 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762844 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762709 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762844 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762790 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762848 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.762969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762866 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762855 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.762969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762916 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762967 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fff690-40c3-413e-98d5-a4955dda45b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.762996 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763027 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0914efbd-e8c2-4055-922b-760acb575c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763103 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763108 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763136 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763181 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0914efbd-e8c2-4055-922b-760acb575c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763211 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763379 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763511 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763562 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763623 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.763843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763769 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fff690-40c3-413e-98d5-a4955dda45b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.763892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.763840 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.764205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.764181 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0914efbd-e8c2-4055-922b-760acb575c85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.765083 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.765059 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0914efbd-e8c2-4055-922b-760acb575c85-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.765381 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.765360 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.765860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.765831 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0914efbd-e8c2-4055-922b-760acb575c85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.766897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.766872 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.816243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.816215 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.816398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.816241 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.816455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.816431 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drt7m\" (UniqueName: \"kubernetes.io/projected/a3fff690-40c3-413e-98d5-a4955dda45b3-kube-api-access-drt7m\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7zm8m\" (UID: \"a3fff690-40c3-413e-98d5-a4955dda45b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.816494 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.816477 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqsv\" (UniqueName: \"kubernetes.io/projected/0914efbd-e8c2-4055-922b-760acb575c85-kube-api-access-gjqsv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-j9dwx\" (UID: \"0914efbd-e8c2-4055-922b-760acb575c85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:27.849366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.849325 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:27.894204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:27.894159 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:28.272792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.272696 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m"] Nov 02 22:36:28.276967 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.276927 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx"] Nov 02 22:36:28.279811 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:36:28.279785 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fff690_40c3_413e_98d5_a4955dda45b3.slice/crio-1e3c29e5728b903986ed0aba5c3a7d152c37a56e68874c98e70795d3c13c49ad WatchSource:0}: Error finding container 1e3c29e5728b903986ed0aba5c3a7d152c37a56e68874c98e70795d3c13c49ad: Status 404 returned error can't find the container with id 1e3c29e5728b903986ed0aba5c3a7d152c37a56e68874c98e70795d3c13c49ad Nov 02 22:36:28.282149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.282119 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.282224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.282207 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.282266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.282235 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.325260 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:36:28.325229 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0914efbd_e8c2_4055_922b_760acb575c85.slice/crio-adbb8b96466bff5c9cbe21239f0c79464405a80c7d7c4b051a476d9d4ae3ee98 WatchSource:0}: Error finding container adbb8b96466bff5c9cbe21239f0c79464405a80c7d7c4b051a476d9d4ae3ee98: Status 404 returned error can't find the container with id adbb8b96466bff5c9cbe21239f0c79464405a80c7d7c4b051a476d9d4ae3ee98 Nov 02 22:36:28.327648 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.327615 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.327761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.327686 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.327761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.327714 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:36:28.518705 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.518666 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" event={"ID":"0914efbd-e8c2-4055-922b-760acb575c85","Type":"ContainerStarted","Data":"c95bf7728c407ff768a873f0fb7f77b143c3eb1b9672b60f9e92568a4726e5fd"} Nov 02 22:36:28.518938 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.518733 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" event={"ID":"0914efbd-e8c2-4055-922b-760acb575c85","Type":"ContainerStarted","Data":"adbb8b96466bff5c9cbe21239f0c79464405a80c7d7c4b051a476d9d4ae3ee98"} Nov 02 22:36:28.520672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.520643 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" event={"ID":"a3fff690-40c3-413e-98d5-a4955dda45b3","Type":"ContainerStarted","Data":"7fa0b983cb2e9c0a8da1d374995bcbf5655ebaeb3878ac3fcbc33e44fd46f65e"} Nov 02 22:36:28.520788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.520697 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" event={"ID":"a3fff690-40c3-413e-98d5-a4955dda45b3","Type":"ContainerStarted","Data":"1e3c29e5728b903986ed0aba5c3a7d152c37a56e68874c98e70795d3c13c49ad"} Nov 02 22:36:28.629083 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.629027 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podStartSLOduration=1.6290097879999998 podStartE2EDuration="1.629009788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:28.626992425 +0000 UTC m=+5228.175242228" watchObservedRunningTime="2026-11-02 22:36:28.629009788 +0000 UTC m=+5228.177259571" Nov 02 22:36:28.735228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.735156 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podStartSLOduration=1.7351404449999999 podStartE2EDuration="1.735140445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:28.734768928 +0000 UTC m=+5228.283018727" watchObservedRunningTime="2026-11-02 22:36:28.735140445 +0000 UTC m=+5228.283390226" Nov 02 22:36:28.852111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.852056 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:28.894872 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:28.894790 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:29.471677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:29.471643 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:29.856832 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:29.856793 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:29.857266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:29.856864 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:29.895851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:29.895814 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:29.896043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:29.895868 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:30.851981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:30.851462 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:30.851981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:30.851553 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:30.895332 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:30.895298 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:30.895711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:30.895372 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:31.854120 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:31.853423 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:31.854120 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:31.853493 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:31.895329 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:31.895286 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:31.895532 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:31.895353 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:32.851988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:32.851417 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:32.851988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:32.851478 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:32.894853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:32.894810 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:32.895042 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:32.894871 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:33.853981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:33.853485 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:33.853981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:33.853555 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:33.894942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:33.894906 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:33.895140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:33.894975 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:34.849969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:34.849919 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:36:34.850185 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:34.850003 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:36:34.895552 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:34.895520 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 02 22:36:34.895942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:34.895578 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 02 22:36:36.442737 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.442699 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:36.443495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.443473 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:36.444384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.444362 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" Nov 02 22:36:36.455560 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.455536 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:36.456201 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.456173 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:36.456939 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:36.456923 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" Nov 02 22:36:37.384492 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.384458 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv"] Nov 02 22:36:37.388139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.388120 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.485330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.485292 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:36:37.485330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.485299 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:36:37.487249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.487229 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv"] Nov 02 22:36:37.502432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.502408 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.502575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.502439 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.502575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.502484 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95lb\" (UniqueName: \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.532006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.531976 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:36:37.532006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.531977 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:36:37.604067 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.604041 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.604233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.604072 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.604233 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.604111 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95lb\" (UniqueName: \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.604711 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.604691 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.606527 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.606503 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.657822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.657754 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95lb\" (UniqueName: \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\") pod \"maas-api-revoke-sub-85237e45-77mxv\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:37.702064 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:37.702037 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:38.064525 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:38.064450 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv"] Nov 02 22:36:38.070004 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:36:38.069975 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47a0d7b_c7d3_41f5_9fc8_1d439aa37045.slice/crio-20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70 WatchSource:0}: Error finding container 20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70: Status 404 returned error can't find the container with id 20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70 Nov 02 22:36:38.610496 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:38.610332 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" event={"ID":"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045","Type":"ContainerStarted","Data":"eab4a5fff6b864b5f91a46da638dfffd22bb6d950eee3ccbcccc068d1393e796"} Nov 02 22:36:38.610496 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:38.610396 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" event={"ID":"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045","Type":"ContainerStarted","Data":"20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70"} Nov 02 22:36:38.718491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:38.718440 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" podStartSLOduration=1.71842536 podStartE2EDuration="1.71842536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:38.716776969 +0000 UTC m=+5238.265026747" watchObservedRunningTime="2026-11-02 22:36:38.71842536 +0000 UTC m=+5238.266675171" Nov 02 22:36:39.617385 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:39.617353 5859 generic.go:356] "Generic (PLEG): container finished" podID="d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" containerID="eab4a5fff6b864b5f91a46da638dfffd22bb6d950eee3ccbcccc068d1393e796" exitCode=0 Nov 02 22:36:39.617788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:39.617444 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" event={"ID":"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045","Type":"ContainerDied","Data":"eab4a5fff6b864b5f91a46da638dfffd22bb6d950eee3ccbcccc068d1393e796"} Nov 02 22:36:40.759615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.759587 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:40.937438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.937358 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\") pod \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " Nov 02 22:36:40.937438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.937411 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\") pod \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " Nov 02 22:36:40.937642 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.937532 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\" (UniqueName: \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\") pod \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\" (UID: \"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045\") " Nov 02 22:36:40.937780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.937759 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca" pod "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" (UID: "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:40.937886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.937872 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:40.939548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.939521 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca" pod "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" (UID: "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:40.939697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:40.939672 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb" pod "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" (UID: "d47a0d7b-c7d3-41f5-9fc8-1d439aa37045"). InnerVolumeSpecName "kube-api-access-k95lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:41.039342 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:41.039314 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95lb\" (UniqueName: \"kubernetes.io/projected/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-kube-api-access-k95lb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:41.039342 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:41.039339 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:41.629199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:41.629165 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" Nov 02 22:36:41.629199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:41.629185 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv" event={"ID":"d47a0d7b-c7d3-41f5-9fc8-1d439aa37045","Type":"ContainerDied","Data":"20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70"} Nov 02 22:36:41.629427 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:41.629238 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e13b3097d42300efd035c634f73297f447fdc53db7f467ceac94b653b5fe70" Nov 02 22:36:49.564313 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.564279 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:49.564689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.564470 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" podUID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" containerName="manager" containerID="cri-o://2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f" gracePeriod=30 Nov 02 22:36:49.813565 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.813540 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:49.922316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.922287 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\") pod \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " Nov 02 22:36:49.922506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.922379 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\") pod \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " Nov 02 22:36:49.922506 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.922451 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\" (UniqueName: \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\") pod \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\" (UID: \"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8\") " Nov 02 22:36:49.925102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.925053 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert" pod "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" (UID: "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:49.925256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.925143 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs" pod "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" (UID: "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:49.925256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:49.925236 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h" pod "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" (UID: "c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8"). InnerVolumeSpecName "kube-api-access-mgb9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:50.024157 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.024123 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgb9h\" (UniqueName: \"kubernetes.io/projected/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-kube-api-access-mgb9h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:50.024157 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.024156 5859 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-webhook-cert\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:50.024325 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.024169 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:50.679619 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.679589 5859 generic.go:356] "Generic (PLEG): container finished" podID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" containerID="2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f" exitCode=0 Nov 02 22:36:50.680061 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.679626 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" event={"ID":"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8","Type":"ContainerDied","Data":"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f"} Nov 02 22:36:50.680061 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.679681 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" Nov 02 22:36:50.680061 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.679690 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-zfcwf" event={"ID":"c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8","Type":"ContainerDied","Data":"2b619358d715ab2bdaebdc88122f46b6bb9fc84dddad76507de070f213fe87d0"} Nov 02 22:36:50.680061 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.679723 5859 scope.go:122] "RemoveContainer" containerID="2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f" Nov 02 22:36:50.688648 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.688624 5859 scope.go:122] "RemoveContainer" containerID="2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f" Nov 02 22:36:50.688910 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:50.688893 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f\": container with ID starting with 2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f not found: ID does not exist" containerID="2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f" Nov 02 22:36:50.688995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.688919 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f"} err="failed to get container status \"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f\": rpc error: code = NotFound desc = could not find container \"2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f\": container with ID starting with 2ea9e21a321c0a49275f871e5c4a98d830907117fa7e9df037c878989cc5358f not found: ID does not exist" Nov 02 22:36:50.833178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.833130 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:50.879288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:50.879267 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-zfcwf"] Nov 02 22:36:52.159734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.159706 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-t7cvb"] Nov 02 22:36:52.160117 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160098 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" containerName="revoke-keys" Nov 02 22:36:52.160117 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160109 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" containerName="revoke-keys" Nov 02 22:36:52.160195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160128 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" containerName="manager" Nov 02 22:36:52.160195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160133 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" containerName="manager" Nov 02 22:36:52.160261 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160215 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" containerName="revoke-keys" Nov 02 22:36:52.160261 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.160225 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" containerName="manager" Nov 02 22:36:52.166163 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.166143 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.215517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.215493 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 22:36:52.241111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.241087 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-metrics-certs\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.241230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.241117 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-webhook-cert\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.241230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.241147 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwpc\" (UniqueName: \"kubernetes.io/projected/3632f233-0d30-47af-afb3-0d59fd3444c9-kube-api-access-kdwpc\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.262394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.262373 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvsgp\"" Nov 02 22:36:52.262513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.262398 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 22:36:52.262513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.262402 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 22:36:52.265012 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.264993 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-t7cvb"] Nov 02 22:36:52.342457 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.342436 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-metrics-certs\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.342548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.342462 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-webhook-cert\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.342548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.342482 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwpc\" (UniqueName: \"kubernetes.io/projected/3632f233-0d30-47af-afb3-0d59fd3444c9-kube-api-access-kdwpc\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.344907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.344883 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-metrics-certs\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.344995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.344940 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3632f233-0d30-47af-afb3-0d59fd3444c9-webhook-cert\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.395452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.395432 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwpc\" (UniqueName: \"kubernetes.io/projected/3632f233-0d30-47af-afb3-0d59fd3444c9-kube-api-access-kdwpc\") pod \"maas-controller-7d468bc7cc-t7cvb\" (UID: \"3632f233-0d30-47af-afb3-0d59fd3444c9\") " pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.479609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.479564 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:52.569376 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.569347 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8" path="/var/lib/kubelet/pods/c74e712b-1a7d-4142-8cb2-c7a20ff3e3b8/volumes" Nov 02 22:36:52.843447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:52.843421 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-t7cvb"] Nov 02 22:36:52.848859 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:36:52.848832 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3632f233_0d30_47af_afb3_0d59fd3444c9.slice/crio-da4cf54e1a5d9725491924817b50d76643157bf71a786cc1080a3aa908c8135d WatchSource:0}: Error finding container da4cf54e1a5d9725491924817b50d76643157bf71a786cc1080a3aa908c8135d: Status 404 returned error can't find the container with id da4cf54e1a5d9725491924817b50d76643157bf71a786cc1080a3aa908c8135d Nov 02 22:36:53.695241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:53.695213 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" event={"ID":"3632f233-0d30-47af-afb3-0d59fd3444c9","Type":"ContainerStarted","Data":"da4cf54e1a5d9725491924817b50d76643157bf71a786cc1080a3aa908c8135d"} Nov 02 22:36:54.709470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:54.709430 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" event={"ID":"3632f233-0d30-47af-afb3-0d59fd3444c9","Type":"ContainerStarted","Data":"455967d30310e2eb2f9e531e9f939dc6794f845f32a4f89ecc90a7e4788d096f"} Nov 02 22:36:54.709866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:54.709518 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:36:54.820212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:36:54.820155 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" podStartSLOduration=1.840082566 podStartE2EDuration="2.820140326s" totalImagesPullingTime="980.05776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:54.816757763 +0000 UTC m=+5254.365007569" watchObservedRunningTime="2026-11-02 22:36:54.820140326 +0000 UTC m=+5254.368390108" Nov 02 22:36:55.947439 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:36:55.947402 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:37:05.716305 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:05.716280 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d468bc7cc-t7cvb" Nov 02 22:37:22.611753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.611715 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x"] Nov 02 22:37:22.616549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.616528 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.667309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.667264 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:37:22.713289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.713256 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:22.713289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.713257 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:37:22.713289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.713302 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:37:22.715313 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.715290 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x"] Nov 02 22:37:22.733524 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.733500 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.733636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.733527 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.733724 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.733704 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dt7x\" (UniqueName: \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.834444 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.834417 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.834591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.834450 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.834591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.834505 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dt7x\" (UniqueName: \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.835089 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.835070 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.836804 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.836786 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.887244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.887221 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dt7x\" (UniqueName: \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\") pod \"maas-api-revoke-sub-f904bff9-xlr6x\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:22.929185 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:22.929161 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:23.292646 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:23.292569 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x"] Nov 02 22:37:23.298402 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:37:23.298367 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4978f6e4_dda6_486e_b956_2a38ea2415c9.slice/crio-d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df WatchSource:0}: Error finding container d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df: Status 404 returned error can't find the container with id d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df Nov 02 22:37:23.864823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:23.864729 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" event={"ID":"4978f6e4-dda6-486e-b956-2a38ea2415c9","Type":"ContainerStarted","Data":"073ed6b4e61acc4b28b29641f5d868a1e77adb48dfb3b7c53c1f1c633ab73a65"} Nov 02 22:37:23.864823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:23.864807 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" event={"ID":"4978f6e4-dda6-486e-b956-2a38ea2415c9","Type":"ContainerStarted","Data":"d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df"} Nov 02 22:37:23.974027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:23.973973 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" podStartSLOduration=1.9739420779999999 podStartE2EDuration="1.973942078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:23.971483729 +0000 UTC m=+5283.519733524" watchObservedRunningTime="2026-11-02 22:37:23.973942078 +0000 UTC m=+5283.522191872" Nov 02 22:37:24.871475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:24.871444 5859 generic.go:356] "Generic (PLEG): container finished" podID="4978f6e4-dda6-486e-b956-2a38ea2415c9" containerID="073ed6b4e61acc4b28b29641f5d868a1e77adb48dfb3b7c53c1f1c633ab73a65" exitCode=0 Nov 02 22:37:24.871863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:24.871534 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" event={"ID":"4978f6e4-dda6-486e-b956-2a38ea2415c9","Type":"ContainerDied","Data":"073ed6b4e61acc4b28b29641f5d868a1e77adb48dfb3b7c53c1f1c633ab73a65"} Nov 02 22:37:25.946237 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:37:25.946205 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:37:26.016187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.016161 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:26.166717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.166630 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\") pod \"4978f6e4-dda6-486e-b956-2a38ea2415c9\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " Nov 02 22:37:26.166717 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.166685 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\" (UniqueName: \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\") pod \"4978f6e4-dda6-486e-b956-2a38ea2415c9\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " Nov 02 22:37:26.166994 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.166731 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\") pod \"4978f6e4-dda6-486e-b956-2a38ea2415c9\" (UID: \"4978f6e4-dda6-486e-b956-2a38ea2415c9\") " Nov 02 22:37:26.167179 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.167154 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca" pod "4978f6e4-dda6-486e-b956-2a38ea2415c9" (UID: "4978f6e4-dda6-486e-b956-2a38ea2415c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:37:26.168921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.168890 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca" pod "4978f6e4-dda6-486e-b956-2a38ea2415c9" (UID: "4978f6e4-dda6-486e-b956-2a38ea2415c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:26.168921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.168905 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x" pod "4978f6e4-dda6-486e-b956-2a38ea2415c9" (UID: "4978f6e4-dda6-486e-b956-2a38ea2415c9"). InnerVolumeSpecName "kube-api-access-5dt7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:26.267703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.267667 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:26.267703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.267700 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dt7x\" (UniqueName: \"kubernetes.io/projected/4978f6e4-dda6-486e-b956-2a38ea2415c9-kube-api-access-5dt7x\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:26.267703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.267709 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4978f6e4-dda6-486e-b956-2a38ea2415c9-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:26.883709 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.883619 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" Nov 02 22:37:26.883709 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.883655 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x" event={"ID":"4978f6e4-dda6-486e-b956-2a38ea2415c9","Type":"ContainerDied","Data":"d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df"} Nov 02 22:37:26.883901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:26.883711 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d4340ae6cf4511ee77f25d57ded6849fb82296ce2916ba898cc563f60c97df" Nov 02 22:37:43.095071 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.095036 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz"] Nov 02 22:37:43.095703 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.095675 5859 cpu_manager.go:414] "containerMap: removing container" podUID="4978f6e4-dda6-486e-b956-2a38ea2415c9" containerName="revoke-keys" Nov 02 22:37:43.095780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.095709 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4978f6e4-dda6-486e-b956-2a38ea2415c9" containerName="revoke-keys" Nov 02 22:37:43.095859 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.095845 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="4978f6e4-dda6-486e-b956-2a38ea2415c9" containerName="revoke-keys" Nov 02 22:37:43.099092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.099074 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222300 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222264 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222440 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222333 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222440 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222385 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222440 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222414 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222583 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222457 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-kube-api-access-tdxwm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222583 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222506 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222583 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222530 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222583 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.222697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.222621 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.296826 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.296794 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz"] Nov 02 22:37:43.323828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.323797 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.323968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.323866 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.323968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.323897 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324042 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-kube-api-access-tdxwm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324086 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324111 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324172 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324210 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324274 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324284 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324491 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324571 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324753 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324731 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.324818 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.324802 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.326380 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.326354 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.326743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.326725 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.376989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.376925 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.377277 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.377258 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/befb4e51-1fca-4d2b-9247-4eebca3cd2f3-kube-api-access-tdxwm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-5chzz\" (UID: \"befb4e51-1fca-4d2b-9247-4eebca3cd2f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.411869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.411850 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:43.832471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.832439 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz"] Nov 02 22:37:43.838096 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:37:43.838069 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefb4e51_1fca_4d2b_9247_4eebca3cd2f3.slice/crio-7c261767abbb5c45e653ba230458ec142401b39b2f192bdf90dde1ece20e07df WatchSource:0}: Error finding container 7c261767abbb5c45e653ba230458ec142401b39b2f192bdf90dde1ece20e07df: Status 404 returned error can't find the container with id 7c261767abbb5c45e653ba230458ec142401b39b2f192bdf90dde1ece20e07df Nov 02 22:37:43.840211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.840181 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:43.840291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.840244 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:43.840291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.840271 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:43.984138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.984098 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" event={"ID":"befb4e51-1fca-4d2b-9247-4eebca3cd2f3","Type":"ContainerStarted","Data":"7b329678a61fcb60012e2398f7e8607e56946b704f896bc1aabff44e459d29f8"} Nov 02 22:37:43.984303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:43.984162 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" event={"ID":"befb4e51-1fca-4d2b-9247-4eebca3cd2f3","Type":"ContainerStarted","Data":"7c261767abbb5c45e653ba230458ec142401b39b2f192bdf90dde1ece20e07df"} Nov 02 22:37:44.094702 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:44.094593 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podStartSLOduration=1.0945797 podStartE2EDuration="1.0945797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:44.091969114 +0000 UTC m=+5303.640218917" watchObservedRunningTime="2026-11-02 22:37:44.0945797 +0000 UTC m=+5303.642829479" Nov 02 22:37:44.427032 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:44.422385 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:44.430110 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:44.430075 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:44.430239 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:44.430143 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:45.429979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:45.424288 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:45.429979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:45.424346 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:46.418974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:46.415264 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:46.418974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:46.415335 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:47.413977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:47.412294 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:47.413977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:47.412350 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:48.420982 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:48.420307 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:48.420982 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:48.420371 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:49.421976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:49.420312 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:49.421976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:49.420372 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:50.412907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:50.412873 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 22:37:50.413144 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:50.412920 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 22:37:52.414388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:52.414335 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:37:52.415029 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:52.414995 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:37:53.467166 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:53.467126 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:37:53.467806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:53.467757 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:37:54.270014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:54.269979 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:54.270676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:54.270654 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:54.271271 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:54.271255 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" Nov 02 22:37:55.945656 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:37:55.945616 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:37:57.218569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.218535 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf"] Nov 02 22:37:57.222417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.222397 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.276107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.276084 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf"] Nov 02 22:37:57.319613 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.319590 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:37:57.319730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.319641 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:57.319730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.319675 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:37:57.319730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.319675 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:37:57.389629 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.389606 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.389755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.389655 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.389755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.389679 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kvg\" (UniqueName: \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.491048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.490984 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.491048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.491036 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kvg\" (UniqueName: \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.491237 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.491140 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.491751 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.491733 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.493264 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.493247 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.544106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.544082 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kvg\" (UniqueName: \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\") pod \"maas-api-revoke-sub-f03629db-zb7xf\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:57.835486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:57.835458 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:37:58.202810 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:58.202737 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf"] Nov 02 22:37:58.208998 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:37:58.208967 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda890c7e6_4b1a_4ffe_b640_cc68fa34c9d2.slice/crio-afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea WatchSource:0}: Error finding container afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea: Status 404 returned error can't find the container with id afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea Nov 02 22:37:59.114670 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:59.114639 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" event={"ID":"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2","Type":"ContainerStarted","Data":"e0681f73c098edeb76e1ed87899f976bcaa9bf68b3b59f124db654cc129a226c"} Nov 02 22:37:59.115072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:59.114691 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" event={"ID":"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2","Type":"ContainerStarted","Data":"afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea"} Nov 02 22:37:59.219921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:37:59.219877 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" podStartSLOduration=2.219863453 podStartE2EDuration="2.219863453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:59.218015784 +0000 UTC m=+5318.766265582" watchObservedRunningTime="2026-11-02 22:37:59.219863453 +0000 UTC m=+5318.768113274" Nov 02 22:38:00.122524 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:00.122481 5859 generic.go:356] "Generic (PLEG): container finished" podID="a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" containerID="e0681f73c098edeb76e1ed87899f976bcaa9bf68b3b59f124db654cc129a226c" exitCode=0 Nov 02 22:38:00.122934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:00.122537 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" event={"ID":"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2","Type":"ContainerDied","Data":"e0681f73c098edeb76e1ed87899f976bcaa9bf68b3b59f124db654cc129a226c"} Nov 02 22:38:01.261156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.261134 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:38:01.429353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.429289 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\") pod \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " Nov 02 22:38:01.429477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.429394 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\" (UniqueName: \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\") pod \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " Nov 02 22:38:01.429477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.429416 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\") pod \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\" (UID: \"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2\") " Nov 02 22:38:01.429677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.429655 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca" pod "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" (UID: "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:01.431410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.431382 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca" pod "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" (UID: "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:01.431510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.431439 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg" pod "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" (UID: "a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2"). InnerVolumeSpecName "kube-api-access-26kvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:01.530875 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.530851 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kvg\" (UniqueName: \"kubernetes.io/projected/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-kube-api-access-26kvg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:01.530875 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.530874 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:01.531032 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:01.530883 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.135178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:02.135145 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" event={"ID":"a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2","Type":"ContainerDied","Data":"afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea"} Nov 02 22:38:02.135178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:02.135171 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf" Nov 02 22:38:02.135390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:02.135203 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe5fee0c39f3ce324221880cf1cf434158e85454e1e4a83021fd7fe6cbf73ea" Nov 02 22:38:13.296190 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.296159 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978"] Nov 02 22:38:13.296605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.296591 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" containerName="revoke-keys" Nov 02 22:38:13.296653 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.296608 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" containerName="revoke-keys" Nov 02 22:38:13.296700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.296691 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" containerName="revoke-keys" Nov 02 22:38:13.299784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.299762 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335412 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335386 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335423 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335452 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335608 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335532 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335643 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335604 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335664 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335714 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335697 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489vk\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-kube-api-access-489vk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335751 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335723 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.335817 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.335804 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.397570 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.397534 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978"] Nov 02 22:38:13.436653 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436615 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489vk\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-kube-api-access-489vk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.436835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436659 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.436835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436700 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.436835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436748 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.436835 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436790 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436842 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436878 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436925 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437101 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.436984 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.437181 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.437239 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.437284 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.437366 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.437549 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.437513 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.439248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.439225 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.439636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.439617 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.489969 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.489932 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.490260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.490237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489vk\" (UniqueName: \"kubernetes.io/projected/e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2-kube-api-access-489vk\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tt978\" (UID: \"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:13.612652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:13.612620 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:14.030762 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.030692 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978"] Nov 02 22:38:14.036022 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:38:14.035996 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ce7041_ca7a_4743_9020_b3f7d7a9fbf2.slice/crio-ec718a650888e0e9aaee19cf2705b669e1d4b0d2b3af726050f5d9626f3e099e WatchSource:0}: Error finding container ec718a650888e0e9aaee19cf2705b669e1d4b0d2b3af726050f5d9626f3e099e: Status 404 returned error can't find the container with id ec718a650888e0e9aaee19cf2705b669e1d4b0d2b3af726050f5d9626f3e099e Nov 02 22:38:14.038170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.038136 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.038256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.038202 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.038256 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.038229 5859 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.201774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.201723 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" event={"ID":"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2","Type":"ContainerStarted","Data":"f0ae6538ee59bf10207eff56b90f977970b697851a93e35318a8a53ee69c78bd"} Nov 02 22:38:14.202010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.201794 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" event={"ID":"e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2","Type":"ContainerStarted","Data":"ec718a650888e0e9aaee19cf2705b669e1d4b0d2b3af726050f5d9626f3e099e"} Nov 02 22:38:14.314151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.314032 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podStartSLOduration=1.3140117820000001 podStartE2EDuration="1.314011782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:14.311284061 +0000 UTC m=+5333.859533856" watchObservedRunningTime="2026-11-02 22:38:14.314011782 +0000 UTC m=+5333.862261564" Nov 02 22:38:14.616062 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:14.616026 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:15.628968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:15.625841 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:15.628968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:15.625909 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:16.613390 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:16.613350 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:16.613683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:16.613653 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:17.617984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.616358 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:17.617984 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.616434 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:17.836860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.836827 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2"] Nov 02 22:38:17.840694 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.840671 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.860077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.860051 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98h5\" (UniqueName: \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.860205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.860099 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.860205 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.860189 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.891934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.891901 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:17.938147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.938113 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:38:17.938311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.938248 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:38:17.938311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.938259 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:38:17.941845 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.941815 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2"] Nov 02 22:38:17.961359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.961332 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98h5\" (UniqueName: \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.961466 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.961379 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.961466 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.961408 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.962061 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.962045 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:17.963720 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:17.963702 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:18.030651 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:18.030622 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98h5\" (UniqueName: \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\") pod \"maas-api-revoke-sub-0165d7d2-8fmf2\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:18.153409 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:18.153345 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:18.546013 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:18.545925 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2"] Nov 02 22:38:18.551382 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:38:18.551352 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1182a16_633a_479a_8dd3_0f10811527bb.slice/crio-b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b WatchSource:0}: Error finding container b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b: Status 404 returned error can't find the container with id b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b Nov 02 22:38:18.626979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:18.623034 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:18.626979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:18.623084 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:19.335524 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:19.335485 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" event={"ID":"d1182a16-633a-479a-8dd3-0f10811527bb","Type":"ContainerStarted","Data":"945db032df8891671f33d848b69830fcfd3c2b19d82c5fcc50b10a55cacce792"} Nov 02 22:38:19.335727 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:19.335543 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" event={"ID":"d1182a16-633a-479a-8dd3-0f10811527bb","Type":"ContainerStarted","Data":"b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b"} Nov 02 22:38:19.444963 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:19.444899 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" podStartSLOduration=2.444880602 podStartE2EDuration="2.444880602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:19.44127074 +0000 UTC m=+5338.989520526" watchObservedRunningTime="2026-11-02 22:38:19.444880602 +0000 UTC m=+5338.993130383" Nov 02 22:38:19.621335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:19.621250 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:19.621587 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:19.621558 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:20.343417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:20.343384 5859 generic.go:356] "Generic (PLEG): container finished" podID="d1182a16-633a-479a-8dd3-0f10811527bb" containerID="945db032df8891671f33d848b69830fcfd3c2b19d82c5fcc50b10a55cacce792" exitCode=0 Nov 02 22:38:20.343837 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:20.343468 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" event={"ID":"d1182a16-633a-479a-8dd3-0f10811527bb","Type":"ContainerDied","Data":"945db032df8891671f33d848b69830fcfd3c2b19d82c5fcc50b10a55cacce792"} Nov 02 22:38:20.612979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:20.612889 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 02 22:38:20.612979 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:20.612935 5859 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 02 22:38:21.483582 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.483561 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:21.493672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.493654 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\") pod \"d1182a16-633a-479a-8dd3-0f10811527bb\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " Nov 02 22:38:21.493771 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.493717 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\" (UniqueName: \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\") pod \"d1182a16-633a-479a-8dd3-0f10811527bb\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " Nov 02 22:38:21.493771 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.493767 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\") pod \"d1182a16-633a-479a-8dd3-0f10811527bb\" (UID: \"d1182a16-633a-479a-8dd3-0f10811527bb\") " Nov 02 22:38:21.494100 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.494073 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca" pod "d1182a16-633a-479a-8dd3-0f10811527bb" (UID: "d1182a16-633a-479a-8dd3-0f10811527bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:21.495742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.495720 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca" pod "d1182a16-633a-479a-8dd3-0f10811527bb" (UID: "d1182a16-633a-479a-8dd3-0f10811527bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:21.495881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.495865 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5" pod "d1182a16-633a-479a-8dd3-0f10811527bb" (UID: "d1182a16-633a-479a-8dd3-0f10811527bb"). InnerVolumeSpecName "kube-api-access-w98h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:21.594991 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.594947 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:21.595115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.594995 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98h5\" (UniqueName: \"kubernetes.io/projected/d1182a16-633a-479a-8dd3-0f10811527bb-kube-api-access-w98h5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:21.595115 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:21.595009 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1182a16-633a-479a-8dd3-0f10811527bb-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:22.228019 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.226720 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:22.228019 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.227213 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:22.228244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.228023 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" Nov 02 22:38:22.357111 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.357083 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" Nov 02 22:38:22.357261 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.357136 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2" event={"ID":"d1182a16-633a-479a-8dd3-0f10811527bb","Type":"ContainerDied","Data":"b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b"} Nov 02 22:38:22.357261 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:22.357177 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06e12c2cbb189e7ce5abacaf2b2a8dd9ff502affd23fbcc5104e769c5ca1d7b" Nov 02 22:38:25.947117 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:38:25.947074 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:38:29.529554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.529520 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq"] Nov 02 22:38:29.530204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.530173 5859 cpu_manager.go:414] "containerMap: removing container" podUID="d1182a16-633a-479a-8dd3-0f10811527bb" containerName="revoke-keys" Nov 02 22:38:29.530204 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.530199 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1182a16-633a-479a-8dd3-0f10811527bb" containerName="revoke-keys" Nov 02 22:38:29.530375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.530314 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1182a16-633a-479a-8dd3-0f10811527bb" containerName="revoke-keys" Nov 02 22:38:29.533793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.533775 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.575145 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.575110 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7rn\" (UniqueName: \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.575290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.575265 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.575352 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.575336 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.629841 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.629810 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:38:29.630033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.629860 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:38:29.630033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.629859 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:29.630033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.629922 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:38:29.632002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.631979 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq"] Nov 02 22:38:29.676526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.676502 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.676652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.676547 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.676652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.676594 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7rn\" (UniqueName: \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.677211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.677189 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.678880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.678861 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.729199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.729176 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7rn\" (UniqueName: \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\") pod \"maas-api-revoke-sub-6b19338f-rj5fq\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:29.845958 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:29.845925 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:30.211329 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:30.211257 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq"] Nov 02 22:38:30.216398 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:38:30.216363 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eaba47_3262_4c3c_ab2a_1a5c59cbc370.slice/crio-23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938 WatchSource:0}: Error finding container 23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938: Status 404 returned error can't find the container with id 23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938 Nov 02 22:38:30.401656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:30.401626 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" event={"ID":"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370","Type":"ContainerStarted","Data":"23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938"} Nov 02 22:38:31.411879 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:31.411844 5859 generic.go:356] "Generic (PLEG): container finished" podID="a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" containerID="069112dede1d35b313376f37ffbf545243db306cb2db1756a4eb1e48b359abfe" exitCode=0 Nov 02 22:38:31.412283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:31.411885 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" event={"ID":"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370","Type":"ContainerDied","Data":"069112dede1d35b313376f37ffbf545243db306cb2db1756a4eb1e48b359abfe"} Nov 02 22:38:32.555598 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.555575 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:32.603126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.603094 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\") pod \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " Nov 02 22:38:32.603282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.603260 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\") pod \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " Nov 02 22:38:32.603337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.603325 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\" (UniqueName: \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\") pod \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\" (UID: \"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370\") " Nov 02 22:38:32.603488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.603444 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca" pod "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" (UID: "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:32.603715 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.603696 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:32.605672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.605648 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn" pod "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" (UID: "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370"). InnerVolumeSpecName "kube-api-access-jj7rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:32.605672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.605660 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca" pod "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" (UID: "a3eaba47-3262-4c3c-ab2a-1a5c59cbc370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:32.705368 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.705293 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:32.705368 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:32.705323 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7rn\" (UniqueName: \"kubernetes.io/projected/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370-kube-api-access-jj7rn\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:33.424920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:33.424877 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" event={"ID":"a3eaba47-3262-4c3c-ab2a-1a5c59cbc370","Type":"ContainerDied","Data":"23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938"} Nov 02 22:38:33.425112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:33.424899 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq" Nov 02 22:38:33.425112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:33.424970 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23355c894ef1f364c3947fb5a8e2d0d57428a7ec0424e6d3284e670ab9d83938" Nov 02 22:38:44.364085 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.364041 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:38:44.365369 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.365340 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" containerName="revoke-keys" Nov 02 22:38:44.365537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.365522 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" containerName="revoke-keys" Nov 02 22:38:44.365791 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.365775 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" containerName="revoke-keys" Nov 02 22:38:44.375232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.375213 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.471245 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.471210 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-58e3ef14-kserve-self-signed-certs\"" Nov 02 22:38:44.471575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.471559 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-58e3ef14-epp-sa-dockercfg-h8c9m\"" Nov 02 22:38:44.480203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.480171 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsg2r\" (UniqueName: \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.480436 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.480413 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.524006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.519761 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:38:44.602615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.602571 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsg2r\" (UniqueName: \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.604731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.604698 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.625656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.625584 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.677927 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.669825 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsg2r\" (UniqueName: \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\") pod \"e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:44.713264 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:44.712992 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:45.102870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:45.102836 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:38:45.122196 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:38:45.122144 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b08a82_7b17_4cf6_b436_8b9e5d45578c.slice/crio-5c816dd714f3930a174f3418fde2478457779fce52180e33cfa23fb36f2afec7 WatchSource:0}: Error finding container 5c816dd714f3930a174f3418fde2478457779fce52180e33cfa23fb36f2afec7: Status 404 returned error can't find the container with id 5c816dd714f3930a174f3418fde2478457779fce52180e33cfa23fb36f2afec7 Nov 02 22:38:45.552354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:45.552264 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" event={"ID":"37b08a82-7b17-4cf6-b436-8b9e5d45578c","Type":"ContainerStarted","Data":"5c816dd714f3930a174f3418fde2478457779fce52180e33cfa23fb36f2afec7"} Nov 02 22:38:47.571033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:47.570998 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" event={"ID":"37b08a82-7b17-4cf6-b436-8b9e5d45578c","Type":"ContainerStarted","Data":"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888"} Nov 02 22:38:47.571476 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:47.571091 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:38:47.679988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:38:47.679927 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podStartSLOduration=1.587010425 podStartE2EDuration="3.679914818s" totalImagesPullingTime="2.092904393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:47.678222546 +0000 UTC m=+5367.226472326" watchObservedRunningTime="2026-11-02 22:38:47.679914818 +0000 UTC m=+5367.228164599" Nov 02 22:38:55.943440 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:38:55.943398 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:39:10.042924 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:10.042887 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2"] Nov 02 22:39:10.088786 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:10.088755 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aedc7c91-cn7m2"] Nov 02 22:39:10.572894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:10.572852 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a147d492-a0e4-4c72-8bfe-a708d8c89516" path="/var/lib/kubelet/pods/a147d492-a0e4-4c72-8bfe-a708d8c89516/volumes" Nov 02 22:39:18.580448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:18.580418 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:39:23.042171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:23.042133 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn"] Nov 02 22:39:23.088707 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:23.088669 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa584d3-548tn"] Nov 02 22:39:24.569656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:24.569622 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f2f1ab-bf51-4a37-a4fd-da07fed19f3d" path="/var/lib/kubelet/pods/60f2f1ab-bf51-4a37-a4fd-da07fed19f3d/volumes" Nov 02 22:39:25.944757 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:39:25.944723 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:39:51.715610 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:51.715575 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:39:55.945200 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:39:55.945158 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:39:58.596244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:39:58.596204 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:40:06.736040 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:06.735981 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" containerID="cri-o://74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888" gracePeriod=45 Nov 02 22:40:08.579655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:08.579617 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:40:18.579658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:18.579618 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:40:18.580080 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:18.579711 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:40:22.537293 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:22.537259 5859 scope.go:122] "RemoveContainer" containerID="d8c33206445f04e168f67473b98f634957387b49dfb549497d2a22712a88e93c" Nov 02 22:40:22.546259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:22.546241 5859 scope.go:122] "RemoveContainer" containerID="a79b3274acea0c9a9d1ae0fadce405d0413b3bac4f14e27f9c557d5ec5b337ce" Nov 02 22:40:25.944795 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:40:25.944763 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:40:28.580469 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:28.580433 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:40:36.979201 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:36.979173 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:40:37.071569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.071539 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\" (UniqueName: \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\") pod \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " Nov 02 22:40:37.071760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.071624 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\" (UniqueName: \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\") pod \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\" (UID: \"37b08a82-7b17-4cf6-b436-8b9e5d45578c\") " Nov 02 22:40:37.073767 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.073734 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs" pod "37b08a82-7b17-4cf6-b436-8b9e5d45578c" (UID: "37b08a82-7b17-4cf6-b436-8b9e5d45578c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:40:37.073882 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.073834 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r" pod "37b08a82-7b17-4cf6-b436-8b9e5d45578c" (UID: "37b08a82-7b17-4cf6-b436-8b9e5d45578c"). InnerVolumeSpecName "kube-api-access-fsg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:40:37.173065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.173033 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsg2r\" (UniqueName: \"kubernetes.io/projected/37b08a82-7b17-4cf6-b436-8b9e5d45578c-kube-api-access-fsg2r\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:37.173065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.173062 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b08a82-7b17-4cf6-b436-8b9e5d45578c-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:37.286998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.286939 5859 generic.go:356] "Generic (PLEG): container finished" podID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerID="74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888" exitCode=0 Nov 02 22:40:37.287188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.287009 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" Nov 02 22:40:37.287188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.287018 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" event={"ID":"37b08a82-7b17-4cf6-b436-8b9e5d45578c","Type":"ContainerDied","Data":"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888"} Nov 02 22:40:37.287188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.287073 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52" event={"ID":"37b08a82-7b17-4cf6-b436-8b9e5d45578c","Type":"ContainerDied","Data":"5c816dd714f3930a174f3418fde2478457779fce52180e33cfa23fb36f2afec7"} Nov 02 22:40:37.287188 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.287096 5859 scope.go:122] "RemoveContainer" containerID="74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888" Nov 02 22:40:37.296244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.296223 5859 scope.go:122] "RemoveContainer" containerID="74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888" Nov 02 22:40:37.296541 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:40:37.296524 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888\": container with ID starting with 74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888 not found: ID does not exist" containerID="74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888" Nov 02 22:40:37.296590 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.296551 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888"} err="failed to get container status \"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888\": rpc error: code = NotFound desc = could not find container \"74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888\": container with ID starting with 74b300e9c20716ce677f82ad587e562ae54f975ba4106ef73d0291b706851888 not found: ID does not exist" Nov 02 22:40:37.441122 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.441043 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:40:37.444783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:37.444760 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-router-scheduler-69d86bb76c-5ph52"] Nov 02 22:40:38.570443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:38.570406 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" path="/var/lib/kubelet/pods/37b08a82-7b17-4cf6-b436-8b9e5d45578c/volumes" Nov 02 22:40:42.224379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.224344 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:40:42.224812 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.224797 5859 cpu_manager.go:414] "containerMap: removing container" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" Nov 02 22:40:42.224861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.224815 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" Nov 02 22:40:42.224933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.224921 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b08a82-7b17-4cf6-b436-8b9e5d45578c" containerName="main" Nov 02 22:40:42.228391 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.228372 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.296181 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.296150 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:40:42.324876 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.324826 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6zv\" (UniqueName: \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.325202 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.325162 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.325491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.325458 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.325667 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.325645 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.325800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.325785 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.325942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.325925 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.326678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.326651 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6faf18\"/\"kube-root-ca.crt\"" Nov 02 22:40:42.327136 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.327012 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6faf18\"/\"openshift-service-ca.crt\"" Nov 02 22:40:42.327362 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.327347 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6faf18\"/\"simulator-single-main-6faf18-kserve-self-signed-certs\"" Nov 02 22:40:42.327516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.327486 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6faf18\"/\"default-dockercfg-f8rsx\"" Nov 02 22:40:42.427586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427552 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.427772 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427645 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6zv\" (UniqueName: \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.427772 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427668 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.427772 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427714 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.427772 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427762 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.428006 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.427791 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.428126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.428107 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.428241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.428219 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.428291 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.428262 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.429991 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.429968 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.430280 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.430261 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.481347 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.481275 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6zv\" (UniqueName: \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\") pod \"simulator-single-main-6faf18-kserve-7595778d8-df8xg\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.540060 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.540024 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:42.908857 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:42.908817 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:40:42.914582 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:40:42.914547 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7513c075_8aad_44be_a09d_ad55f09b862a.slice/crio-cba7c6a0571725ceba7625c2bc77295323a0f50550818a3215843505a909cb5a WatchSource:0}: Error finding container cba7c6a0571725ceba7625c2bc77295323a0f50550818a3215843505a909cb5a: Status 404 returned error can't find the container with id cba7c6a0571725ceba7625c2bc77295323a0f50550818a3215843505a909cb5a Nov 02 22:40:43.335933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:43.335897 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" event={"ID":"7513c075-8aad-44be-a09d-ad55f09b862a","Type":"ContainerStarted","Data":"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2"} Nov 02 22:40:43.335933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:43.335938 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" event={"ID":"7513c075-8aad-44be-a09d-ad55f09b862a","Type":"ContainerStarted","Data":"cba7c6a0571725ceba7625c2bc77295323a0f50550818a3215843505a909cb5a"} Nov 02 22:40:43.444267 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:43.444212 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" podStartSLOduration=1.444197432 podStartE2EDuration="1.444197432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:43.441577103 +0000 UTC m=+5482.989826904" watchObservedRunningTime="2026-11-02 22:40:43.444197432 +0000 UTC m=+5482.992447209" Nov 02 22:40:43.540876 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:43.540838 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:52.541025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:52.540993 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:52.553752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:52.553729 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:53.401018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:40:53.400988 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:40:55.945190 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:40:55.945145 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:41:25.944302 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:41:25.944261 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:41:27.135275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.135240 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d"] Nov 02 22:41:27.140701 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.140678 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.192655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.192621 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d"] Nov 02 22:41:27.236414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.236377 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:41:27.236414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.236378 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:41:27.236633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.236514 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:27.236633 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.236554 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:41:27.278449 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.278405 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnpg\" (UniqueName: \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.278636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.278479 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.278681 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.278628 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.379870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.379827 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.380074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.379919 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnpg\" (UniqueName: \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.380074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.379985 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.380651 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.380618 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.382303 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.382283 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.433326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.433248 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnpg\" (UniqueName: \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\") pod \"maas-api-revoke-sub-188d1d2b-48g6d\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.453938 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.453902 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:27.824827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:27.824786 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d"] Nov 02 22:41:27.830378 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:41:27.830345 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572ad87e_edc2_4545_83b3_a304c9217dbc.slice/crio-77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb WatchSource:0}: Error finding container 77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb: Status 404 returned error can't find the container with id 77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb Nov 02 22:41:28.580045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:28.580002 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" event={"ID":"572ad87e-edc2-4545-83b3-a304c9217dbc","Type":"ContainerStarted","Data":"41e9a381fb46dd6c41a9534b20bbf8454bae2ebed2b5c94d87452719a47c3677"} Nov 02 22:41:28.580460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:28.580065 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" event={"ID":"572ad87e-edc2-4545-83b3-a304c9217dbc","Type":"ContainerStarted","Data":"77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb"} Nov 02 22:41:28.687403 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:28.687350 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" podStartSLOduration=1.6873327169999999 podStartE2EDuration="1.687332717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:41:28.686298784 +0000 UTC m=+5528.234548585" watchObservedRunningTime="2026-11-02 22:41:28.687332717 +0000 UTC m=+5528.235582498" Nov 02 22:41:29.588085 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:29.588040 5859 generic.go:356] "Generic (PLEG): container finished" podID="572ad87e-edc2-4545-83b3-a304c9217dbc" containerID="41e9a381fb46dd6c41a9534b20bbf8454bae2ebed2b5c94d87452719a47c3677" exitCode=0 Nov 02 22:41:29.588531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:29.588136 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" event={"ID":"572ad87e-edc2-4545-83b3-a304c9217dbc","Type":"ContainerDied","Data":"41e9a381fb46dd6c41a9534b20bbf8454bae2ebed2b5c94d87452719a47c3677"} Nov 02 22:41:30.735311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.735284 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:30.814701 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.814667 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\" (UniqueName: \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\") pod \"572ad87e-edc2-4545-83b3-a304c9217dbc\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " Nov 02 22:41:30.814916 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.814712 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\") pod \"572ad87e-edc2-4545-83b3-a304c9217dbc\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " Nov 02 22:41:30.814916 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.814766 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\") pod \"572ad87e-edc2-4545-83b3-a304c9217dbc\" (UID: \"572ad87e-edc2-4545-83b3-a304c9217dbc\") " Nov 02 22:41:30.815201 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.815177 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca" pod "572ad87e-edc2-4545-83b3-a304c9217dbc" (UID: "572ad87e-edc2-4545-83b3-a304c9217dbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:30.816853 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.816828 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca" pod "572ad87e-edc2-4545-83b3-a304c9217dbc" (UID: "572ad87e-edc2-4545-83b3-a304c9217dbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:30.816977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.816935 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg" pod "572ad87e-edc2-4545-83b3-a304c9217dbc" (UID: "572ad87e-edc2-4545-83b3-a304c9217dbc"). InnerVolumeSpecName "kube-api-access-spnpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:30.915894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.915804 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnpg\" (UniqueName: \"kubernetes.io/projected/572ad87e-edc2-4545-83b3-a304c9217dbc-kube-api-access-spnpg\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:30.915894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.915836 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:30.915894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:30.915852 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572ad87e-edc2-4545-83b3-a304c9217dbc-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:31.603232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:31.603197 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" Nov 02 22:41:31.603232 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:31.603221 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d" event={"ID":"572ad87e-edc2-4545-83b3-a304c9217dbc","Type":"ContainerDied","Data":"77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb"} Nov 02 22:41:31.603455 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:31.603269 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d80071a1e9f38a09ae76a2a9a84deccb4846bf03c9d696199280a8e9b944eb" Nov 02 22:41:41.043105 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:41.043062 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv"] Nov 02 22:41:41.089268 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:41.089235 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-77mxv"] Nov 02 22:41:42.569615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:42.569568 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47a0d7b-c7d3-41f5-9fc8-1d439aa37045" path="/var/lib/kubelet/pods/d47a0d7b-c7d3-41f5-9fc8-1d439aa37045/volumes" Nov 02 22:41:43.165457 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:43.165418 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:41:55.946148 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:41:55.946101 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:41:58.185556 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.185495 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" containerName="main" containerID="cri-o://64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2" gracePeriod=45 Nov 02 22:41:58.420360 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.420333 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:41:58.493774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493712 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.493774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493749 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.493774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493774 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\" (UniqueName: \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.494084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493847 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.494084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493869 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.494084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493909 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\") pod \"7513c075-8aad-44be-a09d-ad55f09b862a\" (UID: \"7513c075-8aad-44be-a09d-ad55f09b862a\") " Nov 02 22:41:58.494084 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.493980 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:41:58.494288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.494155 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:41:58.494288 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.494186 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:41:58.494401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.494312 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.494401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.494348 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.494401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.494358 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.496065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.496036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:41:58.496065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.496047 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:58.496203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.496099 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv" pod "7513c075-8aad-44be-a09d-ad55f09b862a" (UID: "7513c075-8aad-44be-a09d-ad55f09b862a"). InnerVolumeSpecName "kube-api-access-sf6zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:58.595620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.595586 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7513c075-8aad-44be-a09d-ad55f09b862a-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.595620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.595613 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6zv\" (UniqueName: \"kubernetes.io/projected/7513c075-8aad-44be-a09d-ad55f09b862a-kube-api-access-sf6zv\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.595620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.595622 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7513c075-8aad-44be-a09d-ad55f09b862a-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:58.749435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.749356 5859 generic.go:356] "Generic (PLEG): container finished" podID="7513c075-8aad-44be-a09d-ad55f09b862a" containerID="64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2" exitCode=0 Nov 02 22:41:58.749435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.749398 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" event={"ID":"7513c075-8aad-44be-a09d-ad55f09b862a","Type":"ContainerDied","Data":"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2"} Nov 02 22:41:58.749435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.749401 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" Nov 02 22:41:58.749435 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.749425 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" event={"ID":"7513c075-8aad-44be-a09d-ad55f09b862a","Type":"ContainerDied","Data":"cba7c6a0571725ceba7625c2bc77295323a0f50550818a3215843505a909cb5a"} Nov 02 22:41:58.749745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.749440 5859 scope.go:122] "RemoveContainer" containerID="64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2" Nov 02 22:41:58.758448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.758421 5859 scope.go:122] "RemoveContainer" containerID="64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2" Nov 02 22:41:58.758694 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:41:58.758675 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2\": container with ID starting with 64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2 not found: ID does not exist" containerID="64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2" Nov 02 22:41:58.758758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.758703 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2"} err="failed to get container status \"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2\": rpc error: code = NotFound desc = could not find container \"64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2\": container with ID starting with 64ea7201a2cdb72083d914aa26f9b910c126f03b702f1e96ee6cfca471a50cd2 not found: ID does not exist" Nov 02 22:41:58.902150 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.902123 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:41:58.948554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:58.948527 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg"] Nov 02 22:41:59.391695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:41:59.391655 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-6faf18/simulator-single-main-6faf18-kserve-7595778d8-df8xg" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:42:00.569377 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:42:00.569337 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" path="/var/lib/kubelet/pods/7513c075-8aad-44be-a09d-ad55f09b862a/volumes" Nov 02 22:42:25.943769 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:42:25.943734 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:42:26.044224 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:42:26.044196 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x"] Nov 02 22:42:26.090241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:42:26.090217 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xlr6x"] Nov 02 22:42:26.569877 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:42:26.569838 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4978f6e4-dda6-486e-b956-2a38ea2415c9" path="/var/lib/kubelet/pods/4978f6e4-dda6-486e-b956-2a38ea2415c9/volumes" Nov 02 22:42:55.943997 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:42:55.943938 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:43:02.042726 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.042689 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf"] Nov 02 22:43:02.089225 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.089192 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-zb7xf"] Nov 02 22:43:02.204773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.204742 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:43:02.205212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205196 5859 cpu_manager.go:414] "containerMap: removing container" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" containerName="main" Nov 02 22:43:02.205212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205212 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" containerName="main" Nov 02 22:43:02.205353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205247 5859 cpu_manager.go:414] "containerMap: removing container" podUID="572ad87e-edc2-4545-83b3-a304c9217dbc" containerName="revoke-keys" Nov 02 22:43:02.205353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205255 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572ad87e-edc2-4545-83b3-a304c9217dbc" containerName="revoke-keys" Nov 02 22:43:02.205353 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205352 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="572ad87e-edc2-4545-83b3-a304c9217dbc" containerName="revoke-keys" Nov 02 22:43:02.205489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.205368 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="7513c075-8aad-44be-a09d-ad55f09b862a" containerName="main" Nov 02 22:43:02.208761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.208741 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.219710 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219683 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.219843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219731 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.219843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219817 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.219940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219860 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2s5d\" (UniqueName: \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.219940 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219925 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.220049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.219974 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.262552 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.262513 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-51a7601a-kserve-self-signed-certs\"" Nov 02 22:43:02.264562 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.264537 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:43:02.320736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320712 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.320920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320749 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.320920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320774 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.320920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320789 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2s5d\" (UniqueName: \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.320920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320825 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.320920 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.320856 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.321244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.321209 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.321244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.321237 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.321355 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.321291 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.323127 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.323103 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.323321 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.323304 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.373632 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.373594 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2s5d\" (UniqueName: \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\") pod \"e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.520424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.520385 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:02.579987 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.579853 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2" path="/var/lib/kubelet/pods/a890c7e6-4b1a-4ffe-b640-cc68fa34c9d2/volumes" Nov 02 22:43:02.897926 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:02.897842 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:43:02.902600 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:43:02.902573 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbef3c73f_9cc9_49aa_9489_c6372ba72a44.slice/crio-8b92ecc0aaa2f772ab57f1426a8a83663178d973275f8d3505aa7bcbc9dbbf8c WatchSource:0}: Error finding container 8b92ecc0aaa2f772ab57f1426a8a83663178d973275f8d3505aa7bcbc9dbbf8c: Status 404 returned error can't find the container with id 8b92ecc0aaa2f772ab57f1426a8a83663178d973275f8d3505aa7bcbc9dbbf8c Nov 02 22:43:03.108349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:03.108315 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" event={"ID":"bef3c73f-9cc9-49aa-9489-c6372ba72a44","Type":"ContainerStarted","Data":"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c"} Nov 02 22:43:03.108739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:03.108362 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" event={"ID":"bef3c73f-9cc9-49aa-9489-c6372ba72a44","Type":"ContainerStarted","Data":"8b92ecc0aaa2f772ab57f1426a8a83663178d973275f8d3505aa7bcbc9dbbf8c"} Nov 02 22:43:03.215894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:03.215790 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" podStartSLOduration=1.215777512 podStartE2EDuration="1.215777512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:03.21411336 +0000 UTC m=+5622.762363139" watchObservedRunningTime="2026-11-02 22:43:03.215777512 +0000 UTC m=+5622.764027349" Nov 02 22:43:03.521461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:03.521381 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:12.520684 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:12.520648 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:12.533035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:12.533013 5859 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:13.241410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:13.241378 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:43:22.048481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:22.048447 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2"] Nov 02 22:43:22.094881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:22.094849 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-8fmf2"] Nov 02 22:43:22.570555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:22.570521 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1182a16-633a-479a-8dd3-0f10811527bb" path="/var/lib/kubelet/pods/d1182a16-633a-479a-8dd3-0f10811527bb/volumes" Nov 02 22:43:22.595700 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:22.595673 5859 scope.go:122] "RemoveContainer" containerID="eab4a5fff6b864b5f91a46da638dfffd22bb6d950eee3ccbcccc068d1393e796" Nov 02 22:43:25.944134 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:43:25.944087 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:43:33.042379 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:33.042342 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq"] Nov 02 22:43:33.088514 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:33.088491 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rj5fq"] Nov 02 22:43:34.569634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:34.569597 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3eaba47-3262-4c3c-ab2a-1a5c59cbc370" path="/var/lib/kubelet/pods/a3eaba47-3262-4c3c-ab2a-1a5c59cbc370/volumes" Nov 02 22:43:55.900338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:43:55.900300 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:43:55.944859 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:43:55.944818 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:44:00.071017 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.070978 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6"] Nov 02 22:44:00.075027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.075009 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.125544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.125514 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gmq\" (UniqueName: \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.125697 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.125681 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.125785 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.125762 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.127851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.127828 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6"] Nov 02 22:44:00.171495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.171406 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:44:00.171495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.171406 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:44:00.171495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.171431 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:44:00.171495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.171479 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:44:00.226621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.226585 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.226796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.226637 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.226843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.226823 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gmq\" (UniqueName: \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.227392 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.227365 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.229072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.229038 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.279841 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.279814 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gmq\" (UniqueName: \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\") pod \"maas-api-revoke-sub-3e31a506-v2cp6\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.387978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.387938 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:00.760140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:00.760056 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6"] Nov 02 22:44:00.765512 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:44:00.765484 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode566d4c3_8c0c_4f77_8e89_d2306265e5f7.slice/crio-69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e WatchSource:0}: Error finding container 69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e: Status 404 returned error can't find the container with id 69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e Nov 02 22:44:01.569186 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:01.569141 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" event={"ID":"e566d4c3-8c0c-4f77-8e89-d2306265e5f7","Type":"ContainerStarted","Data":"a849c537daa5ca72ea7d7244335804c2344f2b85663942d4e576fdac883e1c62"} Nov 02 22:44:01.569822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:01.569243 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" event={"ID":"e566d4c3-8c0c-4f77-8e89-d2306265e5f7","Type":"ContainerStarted","Data":"69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e"} Nov 02 22:44:01.689977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:01.681991 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" podStartSLOduration=1.681972069 podStartE2EDuration="1.681972069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:01.681448056 +0000 UTC m=+5681.229697837" watchObservedRunningTime="2026-11-02 22:44:01.681972069 +0000 UTC m=+5681.230221856" Nov 02 22:44:02.614726 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:02.614669 5859 generic.go:356] "Generic (PLEG): container finished" podID="e566d4c3-8c0c-4f77-8e89-d2306265e5f7" containerID="a849c537daa5ca72ea7d7244335804c2344f2b85663942d4e576fdac883e1c62" exitCode=0 Nov 02 22:44:02.615489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:02.615444 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" event={"ID":"e566d4c3-8c0c-4f77-8e89-d2306265e5f7","Type":"ContainerDied","Data":"a849c537daa5ca72ea7d7244335804c2344f2b85663942d4e576fdac883e1c62"} Nov 02 22:44:03.752421 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.752396 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:03.791460 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.791427 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\" (UniqueName: \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\") pod \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " Nov 02 22:44:03.791622 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.791483 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\") pod \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " Nov 02 22:44:03.791622 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.791600 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\") pod \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\" (UID: \"e566d4c3-8c0c-4f77-8e89-d2306265e5f7\") " Nov 02 22:44:03.791874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.791847 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca" pod "e566d4c3-8c0c-4f77-8e89-d2306265e5f7" (UID: "e566d4c3-8c0c-4f77-8e89-d2306265e5f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:44:03.793578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.793557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca" pod "e566d4c3-8c0c-4f77-8e89-d2306265e5f7" (UID: "e566d4c3-8c0c-4f77-8e89-d2306265e5f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:44:03.793679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.793626 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq" pod "e566d4c3-8c0c-4f77-8e89-d2306265e5f7" (UID: "e566d4c3-8c0c-4f77-8e89-d2306265e5f7"). InnerVolumeSpecName "kube-api-access-76gmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:44:03.893174 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.893078 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gmq\" (UniqueName: \"kubernetes.io/projected/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-kube-api-access-76gmq\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:03.893174 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.893118 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:03.893174 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:03.893134 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e566d4c3-8c0c-4f77-8e89-d2306265e5f7-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:04.626554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:04.626521 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" Nov 02 22:44:04.626740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:04.626565 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6" event={"ID":"e566d4c3-8c0c-4f77-8e89-d2306265e5f7","Type":"ContainerDied","Data":"69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e"} Nov 02 22:44:04.626740 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:04.626608 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ea68fcf0e158c13d99f84a849914780e72823225d0ab1b5886f456baf2071e" Nov 02 22:44:10.920140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:10.920082 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" podUID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" containerName="main" containerID="cri-o://6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c" gracePeriod=45 Nov 02 22:44:11.171638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.171587 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:44:11.258074 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258049 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258161 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258186 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\" (UniqueName: \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258229 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258270 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258314 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\" (UniqueName: \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\") pod \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\" (UID: \"bef3c73f-9cc9-49aa-9489-c6372ba72a44\") " Nov 02 22:44:11.258491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258454 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:44:11.258605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258580 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:44:11.258680 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258599 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:44:11.258742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258696 5859 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-home\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.258742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258713 5859 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tmp-dir\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.258742 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.258725 5859 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-model-cache\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.260372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.260344 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:44:11.260574 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.260552 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:44:11.260636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.260623 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d" pod "bef3c73f-9cc9-49aa-9489-c6372ba72a44" (UID: "bef3c73f-9cc9-49aa-9489-c6372ba72a44"). InnerVolumeSpecName "kube-api-access-k2s5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:44:11.359677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.359655 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2s5d\" (UniqueName: \"kubernetes.io/projected/bef3c73f-9cc9-49aa-9489-c6372ba72a44-kube-api-access-k2s5d\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.359677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.359675 5859 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef3c73f-9cc9-49aa-9489-c6372ba72a44-tls-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.359828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.359684 5859 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef3c73f-9cc9-49aa-9489-c6372ba72a44-dshm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:11.665589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.665562 5859 generic.go:356] "Generic (PLEG): container finished" podID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" containerID="6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c" exitCode=0 Nov 02 22:44:11.665729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.665604 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" Nov 02 22:44:11.665729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.665619 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" event={"ID":"bef3c73f-9cc9-49aa-9489-c6372ba72a44","Type":"ContainerDied","Data":"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c"} Nov 02 22:44:11.665729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.665661 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb" event={"ID":"bef3c73f-9cc9-49aa-9489-c6372ba72a44","Type":"ContainerDied","Data":"8b92ecc0aaa2f772ab57f1426a8a83663178d973275f8d3505aa7bcbc9dbbf8c"} Nov 02 22:44:11.665729 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.665685 5859 scope.go:122] "RemoveContainer" containerID="6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c" Nov 02 22:44:11.675172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.675149 5859 scope.go:122] "RemoveContainer" containerID="6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c" Nov 02 22:44:11.675416 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:44:11.675399 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c\": container with ID starting with 6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c not found: ID does not exist" containerID="6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c" Nov 02 22:44:11.675462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.675424 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c"} err="failed to get container status \"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c\": rpc error: code = NotFound desc = could not find container \"6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c\": container with ID starting with 6763116d61dff1fdc043f9ba78e3fba3124c42f509f6837b00e7c5250518eb9c not found: ID does not exist" Nov 02 22:44:11.817896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.817871 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:44:11.864240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:11.864219 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-51a7601a-kserve-7cb7b797bf-vr7jb"] Nov 02 22:44:12.569171 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:12.569138 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" path="/var/lib/kubelet/pods/bef3c73f-9cc9-49aa-9489-c6372ba72a44/volumes" Nov 02 22:44:22.623121 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:22.623088 5859 scope.go:122] "RemoveContainer" containerID="e0681f73c098edeb76e1ed87899f976bcaa9bf68b3b59f124db654cc129a226c" Nov 02 22:44:22.631398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:22.631381 5859 scope.go:122] "RemoveContainer" containerID="073ed6b4e61acc4b28b29641f5d868a1e77adb48dfb3b7c53c1f1c633ab73a65" Nov 02 22:44:22.641517 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:22.641485 5859 scope.go:122] "RemoveContainer" containerID="945db032df8891671f33d848b69830fcfd3c2b19d82c5fcc50b10a55cacce792" Nov 02 22:44:25.809806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.809774 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd"] Nov 02 22:44:25.810240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810225 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e566d4c3-8c0c-4f77-8e89-d2306265e5f7" containerName="revoke-keys" Nov 02 22:44:25.810285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810242 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e566d4c3-8c0c-4f77-8e89-d2306265e5f7" containerName="revoke-keys" Nov 02 22:44:25.810285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810266 5859 cpu_manager.go:414] "containerMap: removing container" podUID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" containerName="main" Nov 02 22:44:25.810285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810272 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" containerName="main" Nov 02 22:44:25.810382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810345 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e566d4c3-8c0c-4f77-8e89-d2306265e5f7" containerName="revoke-keys" Nov 02 22:44:25.810382 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.810356 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef3c73f-9cc9-49aa-9489-c6372ba72a44" containerName="main" Nov 02 22:44:25.815272 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.815249 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:25.869087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.868401 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd"] Nov 02 22:44:25.911418 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.911388 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:44:25.911591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.911475 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:44:25.911591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.911490 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:44:25.911591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.911523 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:44:25.945016 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:44:25.944988 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:44:25.974230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.974206 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95k8\" (UniqueName: \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:25.974405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.974241 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:25.974405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:25.974330 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.075589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.075566 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95k8\" (UniqueName: \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.075704 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.075601 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.075704 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.075653 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.076361 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.076338 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.077864 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.077844 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.128515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.128495 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95k8\" (UniqueName: \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\") pod \"maas-api-revoke-sub-b0c06767-45xfd\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.428446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.428397 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:26.793652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.793588 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd"] Nov 02 22:44:26.799234 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:44:26.799211 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584aca84_e0b3_4447_bb29_a1abe94b7957.slice/crio-7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01 WatchSource:0}: Error finding container 7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01: Status 404 returned error can't find the container with id 7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01 Nov 02 22:44:26.814167 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:26.814142 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" event={"ID":"584aca84-e0b3-4447-bb29-a1abe94b7957","Type":"ContainerStarted","Data":"7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01"} Nov 02 22:44:27.821292 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:27.821263 5859 generic.go:356] "Generic (PLEG): container finished" podID="584aca84-e0b3-4447-bb29-a1abe94b7957" containerID="839f72150becee56d094b33c0ea3fc3dd7bb223951c03e370b4917d33cc8d437" exitCode=0 Nov 02 22:44:27.821628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:27.821340 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" event={"ID":"584aca84-e0b3-4447-bb29-a1abe94b7957","Type":"ContainerDied","Data":"839f72150becee56d094b33c0ea3fc3dd7bb223951c03e370b4917d33cc8d437"} Nov 02 22:44:28.954138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:28.954116 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:29.002613 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.002586 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\" (UniqueName: \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\") pod \"584aca84-e0b3-4447-bb29-a1abe94b7957\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " Nov 02 22:44:29.002725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.002623 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\") pod \"584aca84-e0b3-4447-bb29-a1abe94b7957\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " Nov 02 22:44:29.002725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.002704 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\") pod \"584aca84-e0b3-4447-bb29-a1abe94b7957\" (UID: \"584aca84-e0b3-4447-bb29-a1abe94b7957\") " Nov 02 22:44:29.003026 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.003003 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca" pod "584aca84-e0b3-4447-bb29-a1abe94b7957" (UID: "584aca84-e0b3-4447-bb29-a1abe94b7957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:44:29.004373 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.004348 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8" pod "584aca84-e0b3-4447-bb29-a1abe94b7957" (UID: "584aca84-e0b3-4447-bb29-a1abe94b7957"). InnerVolumeSpecName "kube-api-access-x95k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:44:29.004611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.004598 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca" pod "584aca84-e0b3-4447-bb29-a1abe94b7957" (UID: "584aca84-e0b3-4447-bb29-a1abe94b7957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:44:29.103497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.103441 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95k8\" (UniqueName: \"kubernetes.io/projected/584aca84-e0b3-4447-bb29-a1abe94b7957-kube-api-access-x95k8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:29.103497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.103462 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:29.103497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.103473 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584aca84-e0b3-4447-bb29-a1abe94b7957-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:44:29.832924 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.832900 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" Nov 02 22:44:29.833087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.832937 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd" event={"ID":"584aca84-e0b3-4447-bb29-a1abe94b7957","Type":"ContainerDied","Data":"7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01"} Nov 02 22:44:29.833087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:29.832994 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3b6965ccc43035034a7f296de4bbe4f3c3dacd2328a97086cfbf044d799c01" Nov 02 22:44:42.313343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.313311 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:44:42.313886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.313749 5859 cpu_manager.go:414] "containerMap: removing container" podUID="584aca84-e0b3-4447-bb29-a1abe94b7957" containerName="revoke-keys" Nov 02 22:44:42.313886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.313760 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584aca84-e0b3-4447-bb29-a1abe94b7957" containerName="revoke-keys" Nov 02 22:44:42.313886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.313833 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="584aca84-e0b3-4447-bb29-a1abe94b7957" containerName="revoke-keys" Nov 02 22:44:42.316932 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.316914 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.367296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.367263 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 22:44:42.413260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.413230 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tdwwv\"" Nov 02 22:44:42.415290 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.415270 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:44:42.430069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.430041 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.430208 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.430189 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t82vm\" (UniqueName: \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.530664 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.530634 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.530790 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.530722 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t82vm\" (UniqueName: \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.531323 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.531305 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.584191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.584167 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t82vm\" (UniqueName: \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\") pod \"payload-pre-processing-787447866d-rq8wv\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.629257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.629224 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:42.996805 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:42.996726 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:44:43.002084 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:44:43.002055 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37e3d90_307d_4efa_ace9_e9416f47f6ce.slice/crio-72a4dbaf1462165e2a7eb2993455f914079098534a34b6741ec8e448e2a30b41 WatchSource:0}: Error finding container 72a4dbaf1462165e2a7eb2993455f914079098534a34b6741ec8e448e2a30b41: Status 404 returned error can't find the container with id 72a4dbaf1462165e2a7eb2993455f914079098534a34b6741ec8e448e2a30b41 Nov 02 22:44:43.908488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:43.908446 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" event={"ID":"c37e3d90-307d-4efa-ace9-e9416f47f6ce","Type":"ContainerStarted","Data":"1d3a7251b88a7e15e63a57a23008dfdae6d2598f81bf72e9ec2252b6509f89d9"} Nov 02 22:44:43.908911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:43.908510 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" event={"ID":"c37e3d90-307d-4efa-ace9-e9416f47f6ce","Type":"ContainerStarted","Data":"72a4dbaf1462165e2a7eb2993455f914079098534a34b6741ec8e448e2a30b41"} Nov 02 22:44:43.908911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:43.908653 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:44.014678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:44.014620 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" podStartSLOduration=2.014603717 podStartE2EDuration="2.014603717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:44.014115183 +0000 UTC m=+5723.562364996" watchObservedRunningTime="2026-11-02 22:44:44.014603717 +0000 UTC m=+5723.562853498" Nov 02 22:44:54.917214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:54.917184 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:44:55.943775 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:44:55.943732 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:44:56.934484 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:56.934451 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:44:56.938448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:56.938423 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:56.992883 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:56.992854 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:44:57.090219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.090189 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84cdk\" (UniqueName: \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.090394 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.090229 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.191482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.191409 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84cdk\" (UniqueName: \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.191482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.191454 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.192210 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.192187 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.244358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.244338 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84cdk\" (UniqueName: \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\") pod \"payload-processing-6f46468f68-knz64\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.252159 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.252135 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.617396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.617364 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:44:57.623349 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:44:57.623323 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc086030_25b7_46d9_b18e_81284efc2db0.slice/crio-c7206faf34a390fb299a1520ad4ca1ad84449a16a846b09f63c483335a23e935 WatchSource:0}: Error finding container c7206faf34a390fb299a1520ad4ca1ad84449a16a846b09f63c483335a23e935: Status 404 returned error can't find the container with id c7206faf34a390fb299a1520ad4ca1ad84449a16a846b09f63c483335a23e935 Nov 02 22:44:57.986874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.986801 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6f46468f68-knz64" event={"ID":"fc086030-25b7-46d9-b18e-81284efc2db0","Type":"ContainerStarted","Data":"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9"} Nov 02 22:44:57.986874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.986860 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:44:57.987102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:57.986877 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6f46468f68-knz64" event={"ID":"fc086030-25b7-46d9-b18e-81284efc2db0","Type":"ContainerStarted","Data":"c7206faf34a390fb299a1520ad4ca1ad84449a16a846b09f63c483335a23e935"} Nov 02 22:44:58.093354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:44:58.093300 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6f46468f68-knz64" podStartSLOduration=2.093287416 podStartE2EDuration="2.093287416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:58.091993809 +0000 UTC m=+5737.640243589" watchObservedRunningTime="2026-11-02 22:44:58.093287416 +0000 UTC m=+5737.641537196" Nov 02 22:45:00.160336 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.160261 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:00.164128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.164101 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.261482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.261449 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:45:00.261482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.261474 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:45:00.261655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.261507 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:45:00.263551 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.263533 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:00.319620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.319598 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.319737 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.319628 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6sr\" (UniqueName: \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.420783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.420728 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.420783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.420760 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6sr\" (UniqueName: \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.421329 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.421312 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.473976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.473929 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6sr\" (UniqueName: \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\") pod \"maas-api-key-cleanup-29894325-d9l6n\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.476734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.476711 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:00.840615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:00.840582 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:00.847249 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:45:00.847223 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2660e9e1_2113_4510_b7c7_ee4538ded652.slice/crio-50d36808fcef774e2a4b517a9854aefb70097c5ff27ac6a29fcf54a9f4a24c22 WatchSource:0}: Error finding container 50d36808fcef774e2a4b517a9854aefb70097c5ff27ac6a29fcf54a9f4a24c22: Status 404 returned error can't find the container with id 50d36808fcef774e2a4b517a9854aefb70097c5ff27ac6a29fcf54a9f4a24c22 Nov 02 22:45:01.012275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:01.012244 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerStarted","Data":"50d36808fcef774e2a4b517a9854aefb70097c5ff27ac6a29fcf54a9f4a24c22"} Nov 02 22:45:02.019081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:02.019007 5859 generic.go:356] "Generic (PLEG): container finished" podID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerID="c795132572eb69ece885678e67c6d428a64af6d9bc0533d85b4e052127f4e651" exitCode=60 Nov 02 22:45:02.019463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:02.019094 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerDied","Data":"c795132572eb69ece885678e67c6d428a64af6d9bc0533d85b4e052127f4e651"} Nov 02 22:45:02.019463 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:02.019265 5859 scope.go:122] "RemoveContainer" containerID="c795132572eb69ece885678e67c6d428a64af6d9bc0533d85b4e052127f4e651" Nov 02 22:45:03.025830 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:03.025772 5859 generic.go:356] "Generic (PLEG): container finished" podID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerID="e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a" exitCode=60 Nov 02 22:45:03.026187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:03.025844 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerDied","Data":"e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a"} Nov 02 22:45:03.026187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:03.025895 5859 scope.go:122] "RemoveContainer" containerID="c795132572eb69ece885678e67c6d428a64af6d9bc0533d85b4e052127f4e651" Nov 02 22:45:03.026187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:03.026087 5859 scope.go:122] "RemoveContainer" containerID="e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a" Nov 02 22:45:03.026324 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:03.026283 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-d9l6n_odh-ai-gateway-infra(2660e9e1-2113-4510-b7c7-ee4538ded652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" Nov 02 22:45:04.035916 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:04.035888 5859 scope.go:122] "RemoveContainer" containerID="e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a" Nov 02 22:45:04.036311 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:04.036138 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-d9l6n_odh-ai-gateway-infra(2660e9e1-2113-4510-b7c7-ee4538ded652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" Nov 02 22:45:09.001827 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:09.001801 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:45:14.563824 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:14.563791 5859 scope.go:122] "RemoveContainer" containerID="e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a" Nov 02 22:45:15.094738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:15.094710 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerStarted","Data":"a7412b0145e71e79e4c3c4a59c4565631d82389c2c5bb7f123e13336a5c6314a"} Nov 02 22:45:15.201077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:15.201032 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" podStartSLOduration=15.201020683 podStartE2EDuration="15.201020683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:15.199604128 +0000 UTC m=+5754.747853914" watchObservedRunningTime="2026-11-02 22:45:15.201020683 +0000 UTC m=+5754.749270463" Nov 02 22:45:16.104239 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:16.104207 5859 generic.go:356] "Generic (PLEG): container finished" podID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerID="a7412b0145e71e79e4c3c4a59c4565631d82389c2c5bb7f123e13336a5c6314a" exitCode=60 Nov 02 22:45:16.104678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:16.104275 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerDied","Data":"a7412b0145e71e79e4c3c4a59c4565631d82389c2c5bb7f123e13336a5c6314a"} Nov 02 22:45:16.104678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:16.104331 5859 scope.go:122] "RemoveContainer" containerID="e233b013c140c06c1eda743663c04f78da8956f4ef153b9f8b6cce496c58c40a" Nov 02 22:45:16.104678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:16.104587 5859 scope.go:122] "RemoveContainer" containerID="a7412b0145e71e79e4c3c4a59c4565631d82389c2c5bb7f123e13336a5c6314a" Nov 02 22:45:16.104812 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:16.104770 5859 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894325-d9l6n_odh-ai-gateway-infra(2660e9e1-2113-4510-b7c7-ee4538ded652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" Nov 02 22:45:16.209114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:16.209089 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:17.259884 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.259857 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:17.361203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.361175 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\" (UniqueName: \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\") pod \"2660e9e1-2113-4510-b7c7-ee4538ded652\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " Nov 02 22:45:17.361343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.361314 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\") pod \"2660e9e1-2113-4510-b7c7-ee4538ded652\" (UID: \"2660e9e1-2113-4510-b7c7-ee4538ded652\") " Nov 02 22:45:17.361644 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.361616 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca" pod "2660e9e1-2113-4510-b7c7-ee4538ded652" (UID: "2660e9e1-2113-4510-b7c7-ee4538ded652"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:17.363289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.363236 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr" pod "2660e9e1-2113-4510-b7c7-ee4538ded652" (UID: "2660e9e1-2113-4510-b7c7-ee4538ded652"). InnerVolumeSpecName "kube-api-access-ch6sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:17.462616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.462584 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660e9e1-2113-4510-b7c7-ee4538ded652-maas-api-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:17.462616 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:17.462611 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6sr\" (UniqueName: \"kubernetes.io/projected/2660e9e1-2113-4510-b7c7-ee4538ded652-kube-api-access-ch6sr\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:18.118425 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.118399 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" Nov 02 22:45:18.118577 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.118400 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n" event={"ID":"2660e9e1-2113-4510-b7c7-ee4538ded652","Type":"ContainerDied","Data":"50d36808fcef774e2a4b517a9854aefb70097c5ff27ac6a29fcf54a9f4a24c22"} Nov 02 22:45:18.118577 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.118523 5859 scope.go:122] "RemoveContainer" containerID="a7412b0145e71e79e4c3c4a59c4565631d82389c2c5bb7f123e13336a5c6314a" Nov 02 22:45:18.271496 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.271469 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:18.317823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.317797 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894325-d9l6n"] Nov 02 22:45:18.569265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:18.569238 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" path="/var/lib/kubelet/pods/2660e9e1-2113-4510-b7c7-ee4538ded652/volumes" Nov 02 22:45:22.673673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:22.673631 5859 scope.go:122] "RemoveContainer" containerID="069112dede1d35b313376f37ffbf545243db306cb2db1756a4eb1e48b359abfe" Nov 02 22:45:23.921066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:23.921032 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:45:24.035776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:24.035748 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:45:25.944215 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:25.944186 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:45:28.921414 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:28.921359 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-6f46468f68-knz64" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" containerName="payload-processing" containerID="cri-o://fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9" gracePeriod=25 Nov 02 22:45:29.000242 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:45:29.000221 5859 logging.go:55] [core] [Channel #10154 SubChannel #10155]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.217:50052", ServerName: "10.134.0.217:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.217:50052: connect: connection refused" Nov 02 22:45:29.036512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.036481 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" podUID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" containerName="payload-pre-processing" containerID="cri-o://1d3a7251b88a7e15e63a57a23008dfdae6d2598f81bf72e9ec2252b6509f89d9" gracePeriod=25 Nov 02 22:45:29.159776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.159751 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:45:29.176904 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.176844 5859 generic.go:356] "Generic (PLEG): container finished" podID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" containerID="1d3a7251b88a7e15e63a57a23008dfdae6d2598f81bf72e9ec2252b6509f89d9" exitCode=0 Nov 02 22:45:29.177020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.176917 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" event={"ID":"c37e3d90-307d-4efa-ace9-e9416f47f6ce","Type":"ContainerDied","Data":"1d3a7251b88a7e15e63a57a23008dfdae6d2598f81bf72e9ec2252b6509f89d9"} Nov 02 22:45:29.178408 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.178381 5859 generic.go:356] "Generic (PLEG): container finished" podID="fc086030-25b7-46d9-b18e-81284efc2db0" containerID="fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9" exitCode=0 Nov 02 22:45:29.178508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.178426 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6f46468f68-knz64" Nov 02 22:45:29.178508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.178435 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6f46468f68-knz64" event={"ID":"fc086030-25b7-46d9-b18e-81284efc2db0","Type":"ContainerDied","Data":"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9"} Nov 02 22:45:29.178508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.178478 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6f46468f68-knz64" event={"ID":"fc086030-25b7-46d9-b18e-81284efc2db0","Type":"ContainerDied","Data":"c7206faf34a390fb299a1520ad4ca1ad84449a16a846b09f63c483335a23e935"} Nov 02 22:45:29.178508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.178506 5859 scope.go:122] "RemoveContainer" containerID="fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9" Nov 02 22:45:29.187488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.187458 5859 scope.go:122] "RemoveContainer" containerID="fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9" Nov 02 22:45:29.187767 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:29.187736 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9\": container with ID starting with fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9 not found: ID does not exist" containerID="fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9" Nov 02 22:45:29.187836 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.187779 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9"} err="failed to get container status \"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9\": rpc error: code = NotFound desc = could not find container \"fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9\": container with ID starting with fe363a3d973936d50d2eb052fa236f86729b5251f3e012a033b4af750d5d1fd9 not found: ID does not exist" Nov 02 22:45:29.271718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.271241 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\" (UniqueName: \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\") pod \"fc086030-25b7-46d9-b18e-81284efc2db0\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " Nov 02 22:45:29.271718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.271380 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\") pod \"fc086030-25b7-46d9-b18e-81284efc2db0\" (UID: \"fc086030-25b7-46d9-b18e-81284efc2db0\") " Nov 02 22:45:29.272400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.272371 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume" pod "fc086030-25b7-46d9-b18e-81284efc2db0" (UID: "fc086030-25b7-46d9-b18e-81284efc2db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:29.274781 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.274757 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk" pod "fc086030-25b7-46d9-b18e-81284efc2db0" (UID: "fc086030-25b7-46d9-b18e-81284efc2db0"). InnerVolumeSpecName "kube-api-access-84cdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:29.292896 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.292877 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:45:29.372615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.372592 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\") pod \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " Nov 02 22:45:29.372712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.372685 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\" (UniqueName: \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\") pod \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\" (UID: \"c37e3d90-307d-4efa-ace9-e9416f47f6ce\") " Nov 02 22:45:29.373022 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.372999 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume" pod "c37e3d90-307d-4efa-ace9-e9416f47f6ce" (UID: "c37e3d90-307d-4efa-ace9-e9416f47f6ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:29.373107 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.373092 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84cdk\" (UniqueName: \"kubernetes.io/projected/fc086030-25b7-46d9-b18e-81284efc2db0-kube-api-access-84cdk\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:29.373156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.373109 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37e3d90-307d-4efa-ace9-e9416f47f6ce-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:29.373156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.373119 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc086030-25b7-46d9-b18e-81284efc2db0-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:29.374662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.374638 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm" pod "c37e3d90-307d-4efa-ace9-e9416f47f6ce" (UID: "c37e3d90-307d-4efa-ace9-e9416f47f6ce"). InnerVolumeSpecName "kube-api-access-t82vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:29.474434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.474383 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t82vm\" (UniqueName: \"kubernetes.io/projected/c37e3d90-307d-4efa-ace9-e9416f47f6ce-kube-api-access-t82vm\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:29.590839 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.590806 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:45:29.594312 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.594284 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-6f46468f68-knz64"] Nov 02 22:45:29.825594 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.825568 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:45:29.825993 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.825981 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" containerName="payload-pre-processing" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.825995 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" containerName="payload-pre-processing" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826007 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826012 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826028 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826033 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826047 5859 cpu_manager.go:414] "containerMap: removing container" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" containerName="payload-processing" Nov 02 22:45:29.826052 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826051 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" containerName="payload-processing" Nov 02 22:45:29.826296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826121 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826133 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" containerName="payload-pre-processing" Nov 02 22:45:29.826296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826139 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:45:29.826296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.826149 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" containerName="payload-processing" Nov 02 22:45:29.829421 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.829403 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:29.881196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881167 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 22:45:29.881196 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881173 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dqz8h\"" Nov 02 22:45:29.881433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881255 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 22:45:29.881433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881258 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 22:45:29.881433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881281 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 02 22:45:29.881433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.881355 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 22:45:29.883276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.883255 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:45:29.980383 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.980354 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:29.980693 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.980409 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:29.980693 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.980525 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:29.980693 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:29.980578 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89mh\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.000489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.000460 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-6f46468f68-knz64" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.217:50052\" within 1s: context deadline exceeded" Nov 02 22:45:30.000580 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:45:30.000513 5859 logging.go:55] [core] [Channel #10154 SubChannel #10155]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.217:50052", ServerName: "10.134.0.217:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.217:50052: operation was canceled" Nov 02 22:45:30.082049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.081987 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.082152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.082057 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.082152 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.082119 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.082273 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.082178 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89mh\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.082854 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.082833 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.082941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.082924 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.084386 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.084364 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.135470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.135440 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89mh\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\") pod \"payload-processing-external-model-5bcd7997-k5v9n\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.142754 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.142728 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:30.186114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.186086 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" event={"ID":"c37e3d90-307d-4efa-ace9-e9416f47f6ce","Type":"ContainerDied","Data":"72a4dbaf1462165e2a7eb2993455f914079098534a34b6741ec8e448e2a30b41"} Nov 02 22:45:30.186306 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.186295 5859 scope.go:122] "RemoveContainer" containerID="1d3a7251b88a7e15e63a57a23008dfdae6d2598f81bf72e9ec2252b6509f89d9" Nov 02 22:45:30.186450 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.186440 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-787447866d-rq8wv" Nov 02 22:45:30.340338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.340265 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:45:30.343306 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.343284 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-787447866d-rq8wv"] Nov 02 22:45:30.528978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.528939 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:45:30.534523 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:45:30.534498 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27b5f51_7665_46b2_9c16_b0aac9814591.slice/crio-97fa105f1d9875665a1a6b5ae11de6927668b0d5b279dbc9ff9be7ee15e8611f WatchSource:0}: Error finding container 97fa105f1d9875665a1a6b5ae11de6927668b0d5b279dbc9ff9be7ee15e8611f: Status 404 returned error can't find the container with id 97fa105f1d9875665a1a6b5ae11de6927668b0d5b279dbc9ff9be7ee15e8611f Nov 02 22:45:30.570976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.570938 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37e3d90-307d-4efa-ace9-e9416f47f6ce" path="/var/lib/kubelet/pods/c37e3d90-307d-4efa-ace9-e9416f47f6ce/volumes" Nov 02 22:45:30.571325 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:30.571310 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc086030-25b7-46d9-b18e-81284efc2db0" path="/var/lib/kubelet/pods/fc086030-25b7-46d9-b18e-81284efc2db0/volumes" Nov 02 22:45:31.197020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:31.196978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" event={"ID":"b27b5f51-7665-46b2-9c16-b0aac9814591","Type":"ContainerStarted","Data":"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286"} Nov 02 22:45:31.197513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:31.197028 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" event={"ID":"b27b5f51-7665-46b2-9c16-b0aac9814591","Type":"ContainerStarted","Data":"97fa105f1d9875665a1a6b5ae11de6927668b0d5b279dbc9ff9be7ee15e8611f"} Nov 02 22:45:31.197513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:31.197161 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:31.307410 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:31.307283 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" podStartSLOduration=2.307266545 podStartE2EDuration="2.307266545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:45:31.306538667 +0000 UTC m=+5770.854788448" watchObservedRunningTime="2026-11-02 22:45:31.307266545 +0000 UTC m=+5770.855516325" Nov 02 22:45:42.209871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:45:42.209843 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:45:55.943690 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:45:55.943652 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:46:25.944081 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:46:25.944044 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:46:31.043286 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:46:31.043251 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d"] Nov 02 22:46:31.089521 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:46:31.089486 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-188d1d2b-48g6d"] Nov 02 22:46:32.571282 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:46:32.571242 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572ad87e-edc2-4545-83b3-a304c9217dbc" path="/var/lib/kubelet/pods/572ad87e-edc2-4545-83b3-a304c9217dbc/volumes" Nov 02 22:46:55.944658 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:46:55.944613 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:47:05.397735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.397662 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c"] Nov 02 22:47:05.398191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.398133 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:47:05.398191 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.398145 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:47:05.398265 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.398215 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2660e9e1-2113-4510-b7c7-ee4538ded652" containerName="cleanup" Nov 02 22:47:05.401452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.401435 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.470855 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.470829 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcz8v\" (UniqueName: \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.471014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.470862 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.471014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.470936 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.498197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.498172 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:47:05.498197 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.498173 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:47:05.498375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.498248 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:47:05.498375 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.498263 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:47:05.500187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.500167 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c"] Nov 02 22:47:05.572194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.572172 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcz8v\" (UniqueName: \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.572287 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.572201 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.572287 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.572221 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.572886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.572859 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.574467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.574446 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.625607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.625583 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcz8v\" (UniqueName: \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\") pod \"maas-api-revoke-sub-e2547622-dxb9c\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:05.714397 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:05.714334 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:06.081784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:06.081756 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c"] Nov 02 22:47:06.086939 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:47:06.086914 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3824e39b_2395_4ba2_b81b_67cbfbfb6734.slice/crio-490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056 WatchSource:0}: Error finding container 490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056: Status 404 returned error can't find the container with id 490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056 Nov 02 22:47:06.730347 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:06.730280 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" event={"ID":"3824e39b-2395-4ba2-b81b-67cbfbfb6734","Type":"ContainerStarted","Data":"4b2a10b29465bd43d6ca042b61d8f23e1b439669acb231b5aebeb1b37d8a8a2a"} Nov 02 22:47:06.730689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:06.730367 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" event={"ID":"3824e39b-2395-4ba2-b81b-67cbfbfb6734","Type":"ContainerStarted","Data":"490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056"} Nov 02 22:47:06.837621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:06.837575 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" podStartSLOduration=1.8375629789999999 podStartE2EDuration="1.837562979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:06.835802388 +0000 UTC m=+5866.384052187" watchObservedRunningTime="2026-11-02 22:47:06.837562979 +0000 UTC m=+5866.385812759" Nov 02 22:47:07.737534 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:07.737506 5859 generic.go:356] "Generic (PLEG): container finished" podID="3824e39b-2395-4ba2-b81b-67cbfbfb6734" containerID="4b2a10b29465bd43d6ca042b61d8f23e1b439669acb231b5aebeb1b37d8a8a2a" exitCode=0 Nov 02 22:47:07.737903 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:07.737562 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" event={"ID":"3824e39b-2395-4ba2-b81b-67cbfbfb6734","Type":"ContainerDied","Data":"4b2a10b29465bd43d6ca042b61d8f23e1b439669acb231b5aebeb1b37d8a8a2a"} Nov 02 22:47:08.883512 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:08.883490 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:09.007059 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.007002 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\") pod \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " Nov 02 22:47:09.007213 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.007089 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\") pod \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " Nov 02 22:47:09.007213 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.007179 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\" (UniqueName: \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\") pod \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\" (UID: \"3824e39b-2395-4ba2-b81b-67cbfbfb6734\") " Nov 02 22:47:09.007411 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.007385 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca" pod "3824e39b-2395-4ba2-b81b-67cbfbfb6734" (UID: "3824e39b-2395-4ba2-b81b-67cbfbfb6734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:47:09.009116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.009093 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca" pod "3824e39b-2395-4ba2-b81b-67cbfbfb6734" (UID: "3824e39b-2395-4ba2-b81b-67cbfbfb6734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:47:09.009216 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.009096 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v" pod "3824e39b-2395-4ba2-b81b-67cbfbfb6734" (UID: "3824e39b-2395-4ba2-b81b-67cbfbfb6734"). InnerVolumeSpecName "kube-api-access-pcz8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:47:09.108660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.108638 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcz8v\" (UniqueName: \"kubernetes.io/projected/3824e39b-2395-4ba2-b81b-67cbfbfb6734-kube-api-access-pcz8v\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:09.108752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.108662 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:09.108752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.108676 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3824e39b-2395-4ba2-b81b-67cbfbfb6734-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:09.749504 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.749479 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" Nov 02 22:47:09.749649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.749508 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c" event={"ID":"3824e39b-2395-4ba2-b81b-67cbfbfb6734","Type":"ContainerDied","Data":"490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056"} Nov 02 22:47:09.749649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:09.749542 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490aec258ec874dc3eab407def43e6ffe9c0612d72401dbbd94b89ac009bc056" Nov 02 22:47:16.874564 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:16.874533 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:47:21.875148 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:21.875083 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" podUID="b27b5f51-7665-46b2-9c16-b0aac9814591" containerName="payload-processing" containerID="cri-o://f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286" gracePeriod=25 Nov 02 22:47:22.121914 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.121890 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:47:22.160722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.160651 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\") pod \"b27b5f51-7665-46b2-9c16-b0aac9814591\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " Nov 02 22:47:22.160722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.160715 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\") pod \"b27b5f51-7665-46b2-9c16-b0aac9814591\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " Nov 02 22:47:22.160914 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.160844 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\") pod \"b27b5f51-7665-46b2-9c16-b0aac9814591\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " Nov 02 22:47:22.160914 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.160887 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\") pod \"b27b5f51-7665-46b2-9c16-b0aac9814591\" (UID: \"b27b5f51-7665-46b2-9c16-b0aac9814591\") " Nov 02 22:47:22.161086 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.161057 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume" pod "b27b5f51-7665-46b2-9c16-b0aac9814591" (UID: "b27b5f51-7665-46b2-9c16-b0aac9814591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:47:22.161257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.161233 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume" pod "b27b5f51-7665-46b2-9c16-b0aac9814591" (UID: "b27b5f51-7665-46b2-9c16-b0aac9814591"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:47:22.161383 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.161363 5859 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-routing-overlay-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:22.161448 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.161383 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b27b5f51-7665-46b2-9c16-b0aac9814591-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:22.162870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.162848 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh" pod "b27b5f51-7665-46b2-9c16-b0aac9814591" (UID: "b27b5f51-7665-46b2-9c16-b0aac9814591"). InnerVolumeSpecName "kube-api-access-c89mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:47:22.162870 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.162861 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials" pod "b27b5f51-7665-46b2-9c16-b0aac9814591" (UID: "b27b5f51-7665-46b2-9c16-b0aac9814591"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:47:22.263010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.262973 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89mh\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-kube-api-access-c89mh\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:22.263010 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.263003 5859 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b27b5f51-7665-46b2-9c16-b0aac9814591-provider-credentials\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:22.832992 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.832888 5859 generic.go:356] "Generic (PLEG): container finished" podID="b27b5f51-7665-46b2-9c16-b0aac9814591" containerID="f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286" exitCode=0 Nov 02 22:47:22.832992 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.832943 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" Nov 02 22:47:22.832992 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.832979 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" event={"ID":"b27b5f51-7665-46b2-9c16-b0aac9814591","Type":"ContainerDied","Data":"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286"} Nov 02 22:47:22.833275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.833029 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n" event={"ID":"b27b5f51-7665-46b2-9c16-b0aac9814591","Type":"ContainerDied","Data":"97fa105f1d9875665a1a6b5ae11de6927668b0d5b279dbc9ff9be7ee15e8611f"} Nov 02 22:47:22.833275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.833054 5859 scope.go:122] "RemoveContainer" containerID="f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286" Nov 02 22:47:22.841861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.841841 5859 scope.go:122] "RemoveContainer" containerID="f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286" Nov 02 22:47:22.842155 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:47:22.842131 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286\": container with ID starting with f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286 not found: ID does not exist" containerID="f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286" Nov 02 22:47:22.842258 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.842160 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286"} err="failed to get container status \"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286\": rpc error: code = NotFound desc = could not find container \"f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286\": container with ID starting with f9a1a8719c040bda149f0d87c38313a1051ffc274efd7075744d2612fad20286 not found: ID does not exist" Nov 02 22:47:22.986373 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.986340 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:47:22.988122 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:22.988096 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-k5v9n"] Nov 02 22:47:24.598359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:24.598320 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27b5f51-7665-46b2-9c16-b0aac9814591" path="/var/lib/kubelet/pods/b27b5f51-7665-46b2-9c16-b0aac9814591/volumes" Nov 02 22:47:25.348970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:25.348867 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:47:25.349235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:25.348977 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:47:25.943292 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:47:25.943259 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:47:38.834359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834325 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx"] Nov 02 22:47:38.834880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834794 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b27b5f51-7665-46b2-9c16-b0aac9814591" containerName="payload-processing" Nov 02 22:47:38.834880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834806 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b5f51-7665-46b2-9c16-b0aac9814591" containerName="payload-processing" Nov 02 22:47:38.834880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834818 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3824e39b-2395-4ba2-b81b-67cbfbfb6734" containerName="revoke-keys" Nov 02 22:47:38.834880 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834824 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3824e39b-2395-4ba2-b81b-67cbfbfb6734" containerName="revoke-keys" Nov 02 22:47:38.835072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834887 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b5f51-7665-46b2-9c16-b0aac9814591" containerName="payload-processing" Nov 02 22:47:38.835072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.834896 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3824e39b-2395-4ba2-b81b-67cbfbfb6734" containerName="revoke-keys" Nov 02 22:47:38.839464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.839444 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:38.895657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.892370 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx"] Nov 02 22:47:38.925861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.925832 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrhd\" (UniqueName: \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:38.926033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.925901 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:38.926033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.926004 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:38.935392 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.935370 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:47:38.935392 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.935368 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:47:38.935544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.935401 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:47:38.935544 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:38.935444 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:47:39.027470 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.027444 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.027595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.027485 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.027595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.027553 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrhd\" (UniqueName: \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.028203 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.028185 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.029630 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.029601 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.080607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.080584 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrhd\" (UniqueName: \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\") pod \"maas-api-revoke-sub-c1e271a9-gw4lx\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.152081 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.152033 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:39.517458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:39.517387 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx"] Nov 02 22:47:39.522792 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:47:39.522763 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d253cdd_1010_4918_8be2_4fc1336e3674.slice/crio-b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781 WatchSource:0}: Error finding container b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781: Status 404 returned error can't find the container with id b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781 Nov 02 22:47:40.016599 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:40.016561 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" event={"ID":"2d253cdd-1010-4918-8be2-4fc1336e3674","Type":"ContainerStarted","Data":"4fdc213ffdf1a2d1047e4cef3ee05f7f279c1269ed0b7767de0b46785faf2f1b"} Nov 02 22:47:40.017031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:40.016609 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" event={"ID":"2d253cdd-1010-4918-8be2-4fc1336e3674","Type":"ContainerStarted","Data":"b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781"} Nov 02 22:47:40.123767 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:40.123719 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" podStartSLOduration=2.123707116 podStartE2EDuration="2.123707116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:47:40.121370257 +0000 UTC m=+5899.669620039" watchObservedRunningTime="2026-11-02 22:47:40.123707116 +0000 UTC m=+5899.671956891" Nov 02 22:47:41.024347 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:41.024314 5859 generic.go:356] "Generic (PLEG): container finished" podID="2d253cdd-1010-4918-8be2-4fc1336e3674" containerID="4fdc213ffdf1a2d1047e4cef3ee05f7f279c1269ed0b7767de0b46785faf2f1b" exitCode=0 Nov 02 22:47:41.024784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:41.024387 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" event={"ID":"2d253cdd-1010-4918-8be2-4fc1336e3674","Type":"ContainerDied","Data":"4fdc213ffdf1a2d1047e4cef3ee05f7f279c1269ed0b7767de0b46785faf2f1b"} Nov 02 22:47:42.171138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.171110 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:42.361723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.361689 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\") pod \"2d253cdd-1010-4918-8be2-4fc1336e3674\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " Nov 02 22:47:42.361907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.361736 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\" (UniqueName: \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\") pod \"2d253cdd-1010-4918-8be2-4fc1336e3674\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " Nov 02 22:47:42.361907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.361826 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\") pod \"2d253cdd-1010-4918-8be2-4fc1336e3674\" (UID: \"2d253cdd-1010-4918-8be2-4fc1336e3674\") " Nov 02 22:47:42.362110 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.362082 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca" pod "2d253cdd-1010-4918-8be2-4fc1336e3674" (UID: "2d253cdd-1010-4918-8be2-4fc1336e3674"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:47:42.362227 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.362206 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:42.363917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.363888 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd" pod "2d253cdd-1010-4918-8be2-4fc1336e3674" (UID: "2d253cdd-1010-4918-8be2-4fc1336e3674"). InnerVolumeSpecName "kube-api-access-fxrhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:47:42.364020 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.363886 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca" pod "2d253cdd-1010-4918-8be2-4fc1336e3674" (UID: "2d253cdd-1010-4918-8be2-4fc1336e3674"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:47:42.463554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.463521 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d253cdd-1010-4918-8be2-4fc1336e3674-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:42.463554 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:42.463551 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrhd\" (UniqueName: \"kubernetes.io/projected/2d253cdd-1010-4918-8be2-4fc1336e3674-kube-api-access-fxrhd\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:47:43.036526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:43.036490 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" Nov 02 22:47:43.036712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:43.036523 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx" event={"ID":"2d253cdd-1010-4918-8be2-4fc1336e3674","Type":"ContainerDied","Data":"b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781"} Nov 02 22:47:43.036712 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:47:43.036574 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b54c283782caf25f56457871231cb2bbad17369d90a28642228d468213efa781" Nov 02 22:47:55.942939 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:47:55.942901 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:48:04.391637 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.391604 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p"] Nov 02 22:48:04.392087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.392054 5859 cpu_manager.go:414] "containerMap: removing container" podUID="2d253cdd-1010-4918-8be2-4fc1336e3674" containerName="revoke-keys" Nov 02 22:48:04.392087 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.392066 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d253cdd-1010-4918-8be2-4fc1336e3674" containerName="revoke-keys" Nov 02 22:48:04.392230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.392187 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d253cdd-1010-4918-8be2-4fc1336e3674" containerName="revoke-keys" Nov 02 22:48:04.395857 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.395837 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.493989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.493939 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:48:04.494142 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.494122 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:48:04.494142 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.494124 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:48:04.494253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.494122 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:48:04.496254 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.496232 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p"] Nov 02 22:48:04.512725 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.512705 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.512823 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.512746 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.512886 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.512841 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfhc\" (UniqueName: \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.613854 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.613829 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.613974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.613861 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.613974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.613891 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfhc\" (UniqueName: \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.614510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.614490 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.616097 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.616074 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.667744 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.667688 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfhc\" (UniqueName: \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\") pod \"maas-api-revoke-sub-2e1f5b5f-4jj4p\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:04.712591 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:04.712571 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:05.078090 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:05.078056 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p"] Nov 02 22:48:05.084460 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:48:05.084430 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929f3fa9_01ed_4db4_a79e_82b86ce32c61.slice/crio-84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb WatchSource:0}: Error finding container 84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb: Status 404 returned error can't find the container with id 84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb Nov 02 22:48:05.210406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:05.210381 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" event={"ID":"929f3fa9-01ed-4db4-a79e-82b86ce32c61","Type":"ContainerStarted","Data":"84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb"} Nov 02 22:48:06.217354 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:06.217286 5859 generic.go:356] "Generic (PLEG): container finished" podID="929f3fa9-01ed-4db4-a79e-82b86ce32c61" containerID="3057a9c5cd3a43241fef712ec6e29b998e268a7c5da5939f174eb08e07b4fdae" exitCode=0 Nov 02 22:48:06.217676 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:06.217360 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" event={"ID":"929f3fa9-01ed-4db4-a79e-82b86ce32c61","Type":"ContainerDied","Data":"3057a9c5cd3a43241fef712ec6e29b998e268a7c5da5939f174eb08e07b4fdae"} Nov 02 22:48:07.357406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.357379 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:07.435433 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.435409 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\" (UniqueName: \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\") pod \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " Nov 02 22:48:07.435573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.435554 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\") pod \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " Nov 02 22:48:07.435623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.435593 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\") pod \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\" (UID: \"929f3fa9-01ed-4db4-a79e-82b86ce32c61\") " Nov 02 22:48:07.435843 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.435821 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca" pod "929f3fa9-01ed-4db4-a79e-82b86ce32c61" (UID: "929f3fa9-01ed-4db4-a79e-82b86ce32c61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:07.435941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.435927 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:07.437495 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.437469 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc" pod "929f3fa9-01ed-4db4-a79e-82b86ce32c61" (UID: "929f3fa9-01ed-4db4-a79e-82b86ce32c61"). InnerVolumeSpecName "kube-api-access-jnfhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:07.437579 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.437469 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca" pod "929f3fa9-01ed-4db4-a79e-82b86ce32c61" (UID: "929f3fa9-01ed-4db4-a79e-82b86ce32c61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:07.536674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.536608 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f3fa9-01ed-4db4-a79e-82b86ce32c61-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:07.536674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:07.536636 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnfhc\" (UniqueName: \"kubernetes.io/projected/929f3fa9-01ed-4db4-a79e-82b86ce32c61-kube-api-access-jnfhc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:08.228372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:08.228338 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" event={"ID":"929f3fa9-01ed-4db4-a79e-82b86ce32c61","Type":"ContainerDied","Data":"84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb"} Nov 02 22:48:08.228372 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:08.228369 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p" Nov 02 22:48:08.228578 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:08.228394 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84719d21eac0803b604e90b441ea1185a5e5dc06caeab3d658e532862326b9cb" Nov 02 22:48:16.838586 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.838547 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6"] Nov 02 22:48:16.840046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.840019 5859 cpu_manager.go:414] "containerMap: removing container" podUID="929f3fa9-01ed-4db4-a79e-82b86ce32c61" containerName="revoke-keys" Nov 02 22:48:16.840212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.840199 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929f3fa9-01ed-4db4-a79e-82b86ce32c61" containerName="revoke-keys" Nov 02 22:48:16.840571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.840555 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="929f3fa9-01ed-4db4-a79e-82b86ce32c61" containerName="revoke-keys" Nov 02 22:48:16.845014 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.844995 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:16.891724 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.891698 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6"] Nov 02 22:48:16.913930 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.913899 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:16.914161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.914142 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8l6\" (UniqueName: \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:16.914253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.914184 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:16.935593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.935571 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:48:16.935593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.935590 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:48:16.935745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.935573 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:48:16.941027 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:16.940995 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:48:17.014673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.014649 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8l6\" (UniqueName: \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.014784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.014678 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.014784 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.014706 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.015328 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.015307 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.017003 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.016975 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.067461 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.067434 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8l6\" (UniqueName: \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\") pod \"maas-api-revoke-sub-ec843018-4vqf6\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.160862 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.160797 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:17.532222 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:17.532105 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6"] Nov 02 22:48:17.538102 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:48:17.538072 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f81c82f_d32f_4856_ae3c_6d62cdce1f35.slice/crio-249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f WatchSource:0}: Error finding container 249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f: Status 404 returned error can't find the container with id 249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f Nov 02 22:48:18.282076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:18.282048 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" event={"ID":"3f81c82f-d32f-4856-ae3c-6d62cdce1f35","Type":"ContainerStarted","Data":"46da49a3d006f6c9f25e27e4c5acaf5f8ef82d5cdef0845d7331034c8eacaefd"} Nov 02 22:48:18.282396 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:18.282108 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" event={"ID":"3f81c82f-d32f-4856-ae3c-6d62cdce1f35","Type":"ContainerStarted","Data":"249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f"} Nov 02 22:48:18.339457 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:48:18.339432 5859 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f81c82f_d32f_4856_ae3c_6d62cdce1f35.slice/crio-conmon-46da49a3d006f6c9f25e27e4c5acaf5f8ef82d5cdef0845d7331034c8eacaefd.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:48:19.289783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.289355 5859 generic.go:356] "Generic (PLEG): container finished" podID="3f81c82f-d32f-4856-ae3c-6d62cdce1f35" containerID="46da49a3d006f6c9f25e27e4c5acaf5f8ef82d5cdef0845d7331034c8eacaefd" exitCode=0 Nov 02 22:48:19.289783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.289551 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" event={"ID":"3f81c82f-d32f-4856-ae3c-6d62cdce1f35","Type":"ContainerDied","Data":"46da49a3d006f6c9f25e27e4c5acaf5f8ef82d5cdef0845d7331034c8eacaefd"} Nov 02 22:48:19.422143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.422119 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:19.537216 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.537190 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\") pod \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " Nov 02 22:48:19.537358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.537228 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\" (UniqueName: \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\") pod \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " Nov 02 22:48:19.537358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.537285 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\") pod \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\" (UID: \"3f81c82f-d32f-4856-ae3c-6d62cdce1f35\") " Nov 02 22:48:19.537709 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.537678 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca" pod "3f81c82f-d32f-4856-ae3c-6d62cdce1f35" (UID: "3f81c82f-d32f-4856-ae3c-6d62cdce1f35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:19.539337 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.539311 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca" pod "3f81c82f-d32f-4856-ae3c-6d62cdce1f35" (UID: "3f81c82f-d32f-4856-ae3c-6d62cdce1f35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:19.539451 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.539378 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6" pod "3f81c82f-d32f-4856-ae3c-6d62cdce1f35" (UID: "3f81c82f-d32f-4856-ae3c-6d62cdce1f35"). InnerVolumeSpecName "kube-api-access-6c8l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:19.638311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.638285 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:19.638311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.638306 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8l6\" (UniqueName: \"kubernetes.io/projected/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-kube-api-access-6c8l6\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:19.638311 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:19.638315 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f81c82f-d32f-4856-ae3c-6d62cdce1f35-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:20.296401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:20.296378 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" Nov 02 22:48:20.296401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:20.296388 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6" event={"ID":"3f81c82f-d32f-4856-ae3c-6d62cdce1f35","Type":"ContainerDied","Data":"249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f"} Nov 02 22:48:20.296815 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:20.296432 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="249d18caf4b58589af4ecd49018cb057e54a89c532f304554b4cd7c1ee18012f" Nov 02 22:48:22.727529 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:22.727498 5859 scope.go:122] "RemoveContainer" containerID="41e9a381fb46dd6c41a9534b20bbf8454bae2ebed2b5c94d87452719a47c3677" Nov 02 22:48:25.942919 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:48:25.942884 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:48:51.633806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.633775 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4"] Nov 02 22:48:51.634321 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.634184 5859 cpu_manager.go:414] "containerMap: removing container" podUID="3f81c82f-d32f-4856-ae3c-6d62cdce1f35" containerName="revoke-keys" Nov 02 22:48:51.634321 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.634196 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f81c82f-d32f-4856-ae3c-6d62cdce1f35" containerName="revoke-keys" Nov 02 22:48:51.634321 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.634291 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f81c82f-d32f-4856-ae3c-6d62cdce1f35" containerName="revoke-keys" Nov 02 22:48:51.637666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.637648 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.688385 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.688355 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:48:51.688505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.688409 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:48:51.688505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.688497 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:48:51.734926 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.734901 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:48:51.737089 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.737067 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4"] Nov 02 22:48:51.742312 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.742293 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.742388 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.742337 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.742444 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.742422 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fp8\" (UniqueName: \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.843270 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.843244 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fp8\" (UniqueName: \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.843421 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.843403 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.843466 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.843455 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.844128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.844109 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.845661 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.845641 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.896181 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.896126 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fp8\" (UniqueName: \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\") pod \"maas-api-revoke-sub-046e1343-chnh4\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:51.950085 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:51.950062 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:52.316360 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:52.316333 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4"] Nov 02 22:48:52.321669 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:48:52.321644 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb999379d_181e_4d2b_acbe_8b1c9791c854.slice/crio-3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6 WatchSource:0}: Error finding container 3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6: Status 404 returned error can't find the container with id 3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6 Nov 02 22:48:52.484168 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:52.484143 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" event={"ID":"b999379d-181e-4d2b-acbe-8b1c9791c854","Type":"ContainerStarted","Data":"3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6"} Nov 02 22:48:53.491003 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:53.490915 5859 generic.go:356] "Generic (PLEG): container finished" podID="b999379d-181e-4d2b-acbe-8b1c9791c854" containerID="3f60650f0c31b6575f2c2de1682bb7aa80fa4cc3cfca72d0d04a0b8365e709b7" exitCode=0 Nov 02 22:48:53.491330 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:53.491004 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" event={"ID":"b999379d-181e-4d2b-acbe-8b1c9791c854","Type":"ContainerDied","Data":"3f60650f0c31b6575f2c2de1682bb7aa80fa4cc3cfca72d0d04a0b8365e709b7"} Nov 02 22:48:54.624522 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.624495 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:54.768113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.768046 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\" (UniqueName: \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\") pod \"b999379d-181e-4d2b-acbe-8b1c9791c854\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " Nov 02 22:48:54.768113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.768079 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\") pod \"b999379d-181e-4d2b-acbe-8b1c9791c854\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " Nov 02 22:48:54.768113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.768101 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\") pod \"b999379d-181e-4d2b-acbe-8b1c9791c854\" (UID: \"b999379d-181e-4d2b-acbe-8b1c9791c854\") " Nov 02 22:48:54.768476 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.768453 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca" pod "b999379d-181e-4d2b-acbe-8b1c9791c854" (UID: "b999379d-181e-4d2b-acbe-8b1c9791c854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:48:54.770059 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.770036 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca" pod "b999379d-181e-4d2b-acbe-8b1c9791c854" (UID: "b999379d-181e-4d2b-acbe-8b1c9791c854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:48:54.770157 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.770138 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8" pod "b999379d-181e-4d2b-acbe-8b1c9791c854" (UID: "b999379d-181e-4d2b-acbe-8b1c9791c854"). InnerVolumeSpecName "kube-api-access-v9fp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:48:54.869405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.869384 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fp8\" (UniqueName: \"kubernetes.io/projected/b999379d-181e-4d2b-acbe-8b1c9791c854-kube-api-access-v9fp8\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:54.869405 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.869404 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:54.869548 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:54.869415 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b999379d-181e-4d2b-acbe-8b1c9791c854-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:48:55.502351 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:55.502321 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" Nov 02 22:48:55.502508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:55.502369 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4" event={"ID":"b999379d-181e-4d2b-acbe-8b1c9791c854","Type":"ContainerDied","Data":"3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6"} Nov 02 22:48:55.502508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:48:55.502447 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a440bd16f44dbc9d87fb4b4cbb219f115283925abb4ef139f6d476136c8b7b6" Nov 02 22:48:55.943690 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:48:55.943656 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:49:02.405275 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.405246 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz"] Nov 02 22:49:02.406673 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.406637 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b999379d-181e-4d2b-acbe-8b1c9791c854" containerName="revoke-keys" Nov 02 22:49:02.406869 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.406852 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b999379d-181e-4d2b-acbe-8b1c9791c854" containerName="revoke-keys" Nov 02 22:49:02.407307 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.407290 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b999379d-181e-4d2b-acbe-8b1c9791c854" containerName="revoke-keys" Nov 02 22:49:02.411833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.411813 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.462818 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.462784 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz"] Nov 02 22:49:02.506459 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.506435 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:49:02.506588 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.506483 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:49:02.506588 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.506515 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:49:02.508211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.508196 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:49:02.531246 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.531221 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgk7s\" (UniqueName: \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.531327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.531257 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.531386 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.531373 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.632851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.632830 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgk7s\" (UniqueName: \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.632985 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.632863 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.632985 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.632911 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.633542 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.633522 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.635231 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.635204 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.686253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.686195 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgk7s\" (UniqueName: \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\") pod \"maas-api-revoke-sub-d3fba056-jnqtz\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:02.729235 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:02.729214 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:03.101416 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:03.101386 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz"] Nov 02 22:49:03.106024 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:49:03.105998 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027621f8_0c60_4c9a_b98c_1896046a7bad.slice/crio-c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4 WatchSource:0}: Error finding container c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4: Status 404 returned error can't find the container with id c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4 Nov 02 22:49:03.549658 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:03.549629 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" event={"ID":"027621f8-0c60-4c9a-b98c-1896046a7bad","Type":"ContainerStarted","Data":"7b7b6426c9dda7ec15a9d0f28704349b7f60ceeaf3e81e98eb90f4ad81b8379a"} Nov 02 22:49:03.550109 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:03.549682 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" event={"ID":"027621f8-0c60-4c9a-b98c-1896046a7bad","Type":"ContainerStarted","Data":"c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4"} Nov 02 22:49:03.702135 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:03.702068 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" podStartSLOduration=1.702055047 podStartE2EDuration="1.702055047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:03.654928358 +0000 UTC m=+5983.203178157" watchObservedRunningTime="2026-11-02 22:49:03.702055047 +0000 UTC m=+5983.250304828" Nov 02 22:49:04.042342 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:04.042313 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6"] Nov 02 22:49:04.044418 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:04.044394 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v2cp6"] Nov 02 22:49:04.556467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:04.556437 5859 generic.go:356] "Generic (PLEG): container finished" podID="027621f8-0c60-4c9a-b98c-1896046a7bad" containerID="7b7b6426c9dda7ec15a9d0f28704349b7f60ceeaf3e81e98eb90f4ad81b8379a" exitCode=0 Nov 02 22:49:04.556808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:04.556491 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" event={"ID":"027621f8-0c60-4c9a-b98c-1896046a7bad","Type":"ContainerDied","Data":"7b7b6426c9dda7ec15a9d0f28704349b7f60ceeaf3e81e98eb90f4ad81b8379a"} Nov 02 22:49:04.570114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:04.570084 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e566d4c3-8c0c-4f77-8e89-d2306265e5f7" path="/var/lib/kubelet/pods/e566d4c3-8c0c-4f77-8e89-d2306265e5f7/volumes" Nov 02 22:49:05.747314 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.747286 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:05.866975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.863324 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\") pod \"027621f8-0c60-4c9a-b98c-1896046a7bad\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " Nov 02 22:49:05.866975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.863449 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\" (UniqueName: \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\") pod \"027621f8-0c60-4c9a-b98c-1896046a7bad\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " Nov 02 22:49:05.866975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.863500 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\") pod \"027621f8-0c60-4c9a-b98c-1896046a7bad\" (UID: \"027621f8-0c60-4c9a-b98c-1896046a7bad\") " Nov 02 22:49:05.866975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.864290 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca" pod "027621f8-0c60-4c9a-b98c-1896046a7bad" (UID: "027621f8-0c60-4c9a-b98c-1896046a7bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:05.895206 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.881351 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca" pod "027621f8-0c60-4c9a-b98c-1896046a7bad" (UID: "027621f8-0c60-4c9a-b98c-1896046a7bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:05.895206 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.881389 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s" pod "027621f8-0c60-4c9a-b98c-1896046a7bad" (UID: "027621f8-0c60-4c9a-b98c-1896046a7bad"). InnerVolumeSpecName "kube-api-access-sgk7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:05.988974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.988613 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:05.988974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.988652 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgk7s\" (UniqueName: \"kubernetes.io/projected/027621f8-0c60-4c9a-b98c-1896046a7bad-kube-api-access-sgk7s\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:05.988974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:05.988668 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027621f8-0c60-4c9a-b98c-1896046a7bad-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:06.621976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:06.621087 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" event={"ID":"027621f8-0c60-4c9a-b98c-1896046a7bad","Type":"ContainerDied","Data":"c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4"} Nov 02 22:49:06.621976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:06.621147 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5ecee759ddc61eacb022a9babdd99173963972e4135cfc9fb50a5918e466ad4" Nov 02 22:49:06.621976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:06.621213 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz" Nov 02 22:49:07.487123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:07.487077 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:49:07.487656 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:07.487145 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:49:22.491970 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:22.491890 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:49:22.492515 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:22.492430 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:49:22.503292 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:22.503097 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:49:22.503292 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:22.503165 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:49:25.943527 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:49:25.943492 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:49:30.042333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:30.042296 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd"] Nov 02 22:49:30.088589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:30.088558 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c06767-45xfd"] Nov 02 22:49:30.569678 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:30.569643 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584aca84-e0b3-4447-bb29-a1abe94b7957" path="/var/lib/kubelet/pods/584aca84-e0b3-4447-bb29-a1abe94b7957/volumes" Nov 02 22:49:36.828194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.828160 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff"] Nov 02 22:49:36.828657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.828576 5859 cpu_manager.go:414] "containerMap: removing container" podUID="027621f8-0c60-4c9a-b98c-1896046a7bad" containerName="revoke-keys" Nov 02 22:49:36.828657 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.828587 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027621f8-0c60-4c9a-b98c-1896046a7bad" containerName="revoke-keys" Nov 02 22:49:36.828734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.828663 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="027621f8-0c60-4c9a-b98c-1896046a7bad" containerName="revoke-keys" Nov 02 22:49:36.832557 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.832537 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:36.882933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.882904 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:49:36.923865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.923842 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:36.923990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.923887 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:36.923990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.923974 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgz7\" (UniqueName: \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:36.929143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.929126 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:49:36.929143 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.929135 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:49:36.929326 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.929312 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:49:36.931472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:36.931456 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff"] Nov 02 22:49:37.025211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.025176 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.025338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.025238 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.025338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.025276 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgz7\" (UniqueName: \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.025873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.025854 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.027537 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.027519 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.078417 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.078366 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgz7\" (UniqueName: \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\") pod \"maas-api-revoke-sub-ae594979-vmrff\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.145343 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.145323 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:37.514243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.514178 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff"] Nov 02 22:49:37.519635 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:49:37.519607 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10bc770_124c_4384_b084_ebee7eed1de0.slice/crio-33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29 WatchSource:0}: Error finding container 33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29: Status 404 returned error can't find the container with id 33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29 Nov 02 22:49:37.826219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.826189 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" event={"ID":"a10bc770-124c-4384-b084-ebee7eed1de0","Type":"ContainerStarted","Data":"75b6d17fe595828854308b30c3c861ee633070f7d75e91336a295bb252452533"} Nov 02 22:49:37.826335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.826250 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" event={"ID":"a10bc770-124c-4384-b084-ebee7eed1de0","Type":"ContainerStarted","Data":"33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29"} Nov 02 22:49:37.932909 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:37.932869 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" podStartSLOduration=1.932856583 podStartE2EDuration="1.932856583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:49:37.93110387 +0000 UTC m=+6017.479353675" watchObservedRunningTime="2026-11-02 22:49:37.932856583 +0000 UTC m=+6017.481106384" Nov 02 22:49:38.832695 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:38.832662 5859 generic.go:356] "Generic (PLEG): container finished" podID="a10bc770-124c-4384-b084-ebee7eed1de0" containerID="75b6d17fe595828854308b30c3c861ee633070f7d75e91336a295bb252452533" exitCode=0 Nov 02 22:49:38.832878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:38.832749 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" event={"ID":"a10bc770-124c-4384-b084-ebee7eed1de0","Type":"ContainerDied","Data":"75b6d17fe595828854308b30c3c861ee633070f7d75e91336a295bb252452533"} Nov 02 22:49:39.971065 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:39.971045 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:40.050718 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.050694 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\") pod \"a10bc770-124c-4384-b084-ebee7eed1de0\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " Nov 02 22:49:40.050833 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.050729 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\" (UniqueName: \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\") pod \"a10bc770-124c-4384-b084-ebee7eed1de0\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " Nov 02 22:49:40.050876 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.050833 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\") pod \"a10bc770-124c-4384-b084-ebee7eed1de0\" (UID: \"a10bc770-124c-4384-b084-ebee7eed1de0\") " Nov 02 22:49:40.051135 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.051114 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca" pod "a10bc770-124c-4384-b084-ebee7eed1de0" (UID: "a10bc770-124c-4384-b084-ebee7eed1de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:40.051244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.051230 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:40.052620 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.052598 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca" pod "a10bc770-124c-4384-b084-ebee7eed1de0" (UID: "a10bc770-124c-4384-b084-ebee7eed1de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:40.052734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.052719 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7" pod "a10bc770-124c-4384-b084-ebee7eed1de0" (UID: "a10bc770-124c-4384-b084-ebee7eed1de0"). InnerVolumeSpecName "kube-api-access-szgz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:40.152238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.152183 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10bc770-124c-4384-b084-ebee7eed1de0-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:40.152238 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.152204 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgz7\" (UniqueName: \"kubernetes.io/projected/a10bc770-124c-4384-b084-ebee7eed1de0-kube-api-access-szgz7\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:40.844319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.844253 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" event={"ID":"a10bc770-124c-4384-b084-ebee7eed1de0","Type":"ContainerDied","Data":"33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29"} Nov 02 22:49:40.844319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.844302 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33da3575b3942c18b9d0b7553078938b642541bd7488dd9fd7ff578f79f04f29" Nov 02 22:49:40.844319 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:40.844268 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff" Nov 02 22:49:56.074336 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:49:56.074293 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:49:58.519049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.519018 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz"] Nov 02 22:49:58.519452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.519418 5859 cpu_manager.go:414] "containerMap: removing container" podUID="a10bc770-124c-4384-b084-ebee7eed1de0" containerName="revoke-keys" Nov 02 22:49:58.519452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.519429 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10bc770-124c-4384-b084-ebee7eed1de0" containerName="revoke-keys" Nov 02 22:49:58.519543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.519535 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10bc770-124c-4384-b084-ebee7eed1de0" containerName="revoke-keys" Nov 02 22:49:58.523025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.523004 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.573045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.573016 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:49:58.619501 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.619475 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:49:58.619665 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.619522 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:49:58.619665 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.619558 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:49:58.621682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.621661 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz"] Nov 02 22:49:58.672438 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.672415 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.672555 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.672459 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.672615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.672589 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5rj\" (UniqueName: \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.773756 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.773692 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5rj\" (UniqueName: \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.773883 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.773785 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.773883 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.773827 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.774488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.774466 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.776024 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.776000 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.828351 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.828324 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5rj\" (UniqueName: \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\") pod \"maas-api-revoke-sub-a4aef7d7-x54mz\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:58.835996 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:58.835976 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:49:59.203057 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:59.203027 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz"] Nov 02 22:49:59.208721 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:49:59.208695 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e5d064_bb03_4ff2_9114_683434a7a88c.slice/crio-53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4 WatchSource:0}: Error finding container 53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4: Status 404 returned error can't find the container with id 53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4 Nov 02 22:49:59.971474 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:59.971443 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" event={"ID":"78e5d064-bb03-4ff2-9114-683434a7a88c","Type":"ContainerStarted","Data":"2079451c37c3a8595fc2acfc667b76e088504b8e5b91d15a6c636e080518ccce"} Nov 02 22:49:59.971828 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:49:59.971495 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" event={"ID":"78e5d064-bb03-4ff2-9114-683434a7a88c","Type":"ContainerStarted","Data":"53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4"} Nov 02 22:50:00.080488 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:00.080437 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" podStartSLOduration=2.080422965 podStartE2EDuration="2.080422965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:50:00.077600106 +0000 UTC m=+6039.625849913" watchObservedRunningTime="2026-11-02 22:50:00.080422965 +0000 UTC m=+6039.628672746" Nov 02 22:50:00.978082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:00.978043 5859 generic.go:356] "Generic (PLEG): container finished" podID="78e5d064-bb03-4ff2-9114-683434a7a88c" containerID="2079451c37c3a8595fc2acfc667b76e088504b8e5b91d15a6c636e080518ccce" exitCode=0 Nov 02 22:50:00.978508 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:00.978105 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" event={"ID":"78e5d064-bb03-4ff2-9114-683434a7a88c","Type":"ContainerDied","Data":"2079451c37c3a8595fc2acfc667b76e088504b8e5b91d15a6c636e080518ccce"} Nov 02 22:50:02.119183 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.119161 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:50:02.206645 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.206621 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\") pod \"78e5d064-bb03-4ff2-9114-683434a7a88c\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " Nov 02 22:50:02.206756 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.206675 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\" (UniqueName: \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\") pod \"78e5d064-bb03-4ff2-9114-683434a7a88c\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " Nov 02 22:50:02.206806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.206760 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\") pod \"78e5d064-bb03-4ff2-9114-683434a7a88c\" (UID: \"78e5d064-bb03-4ff2-9114-683434a7a88c\") " Nov 02 22:50:02.207025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.207004 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca" pod "78e5d064-bb03-4ff2-9114-683434a7a88c" (UID: "78e5d064-bb03-4ff2-9114-683434a7a88c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:02.208645 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.208620 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca" pod "78e5d064-bb03-4ff2-9114-683434a7a88c" (UID: "78e5d064-bb03-4ff2-9114-683434a7a88c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:02.208773 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.208753 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj" pod "78e5d064-bb03-4ff2-9114-683434a7a88c" (UID: "78e5d064-bb03-4ff2-9114-683434a7a88c"). InnerVolumeSpecName "kube-api-access-9r5rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:02.308301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.308241 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5rj\" (UniqueName: \"kubernetes.io/projected/78e5d064-bb03-4ff2-9114-683434a7a88c-kube-api-access-9r5rj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:02.308301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.308268 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:02.308301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.308281 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e5d064-bb03-4ff2-9114-683434a7a88c-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:02.991842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.991765 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" event={"ID":"78e5d064-bb03-4ff2-9114-683434a7a88c","Type":"ContainerDied","Data":"53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4"} Nov 02 22:50:02.991842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.991783 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz" Nov 02 22:50:02.991842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:02.991836 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53fbb38499db2d4712c9c432e2cda312102dd819e23cdb8845eb6719a76303c4" Nov 02 22:50:19.480975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.478235 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns"] Nov 02 22:50:19.480975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.478932 5859 cpu_manager.go:414] "containerMap: removing container" podUID="78e5d064-bb03-4ff2-9114-683434a7a88c" containerName="revoke-keys" Nov 02 22:50:19.480975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.478972 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e5d064-bb03-4ff2-9114-683434a7a88c" containerName="revoke-keys" Nov 02 22:50:19.480975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.479100 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e5d064-bb03-4ff2-9114-683434a7a88c" containerName="revoke-keys" Nov 02 22:50:19.483477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.482850 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.549133 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.549051 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns"] Nov 02 22:50:19.579333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.579304 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:50:19.579752 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.579733 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:50:19.580002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.579986 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:50:19.603493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.603201 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:50:19.605755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.605718 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.627981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.622356 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5xj\" (UniqueName: \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.627981 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.622644 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.774046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.731481 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.774046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.731554 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5xj\" (UniqueName: \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.774046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.731645 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.774046 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.732895 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.905972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.861111 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:19.905972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:19.898898 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5xj\" (UniqueName: \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\") pod \"maas-api-revoke-sub-79805ca9-n9gns\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:20.124744 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:20.124700 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:20.795116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:20.793449 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns"] Nov 02 22:50:21.118278 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:21.118242 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" event={"ID":"07029a1f-f4a5-4f57-96a7-0574376c9822","Type":"ContainerStarted","Data":"1281964289a2b56b43fb5959866d24d358a7c73243c7c69adbac64beed683e68"} Nov 02 22:50:22.125165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:22.125083 5859 generic.go:356] "Generic (PLEG): container finished" podID="07029a1f-f4a5-4f57-96a7-0574376c9822" containerID="061dc136660409f373fdd98ea6daa87aa3e081e1a7da5991f0fbeba16b4bc82c" exitCode=0 Nov 02 22:50:22.125531 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:22.125152 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" event={"ID":"07029a1f-f4a5-4f57-96a7-0574376c9822","Type":"ContainerDied","Data":"061dc136660409f373fdd98ea6daa87aa3e081e1a7da5991f0fbeba16b4bc82c"} Nov 02 22:50:22.787568 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:22.787536 5859 scope.go:122] "RemoveContainer" containerID="a849c537daa5ca72ea7d7244335804c2344f2b85663942d4e576fdac883e1c62" Nov 02 22:50:23.257365 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.257341 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:23.300248 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.300225 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\") pod \"07029a1f-f4a5-4f57-96a7-0574376c9822\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " Nov 02 22:50:23.300346 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.300295 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\") pod \"07029a1f-f4a5-4f57-96a7-0574376c9822\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " Nov 02 22:50:23.300346 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.300331 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\" (UniqueName: \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\") pod \"07029a1f-f4a5-4f57-96a7-0574376c9822\" (UID: \"07029a1f-f4a5-4f57-96a7-0574376c9822\") " Nov 02 22:50:23.300604 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.300580 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca" pod "07029a1f-f4a5-4f57-96a7-0574376c9822" (UID: "07029a1f-f4a5-4f57-96a7-0574376c9822"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:23.300768 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.300744 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:23.302315 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.302293 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca" pod "07029a1f-f4a5-4f57-96a7-0574376c9822" (UID: "07029a1f-f4a5-4f57-96a7-0574376c9822"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:23.302403 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.302310 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj" pod "07029a1f-f4a5-4f57-96a7-0574376c9822" (UID: "07029a1f-f4a5-4f57-96a7-0574376c9822"). InnerVolumeSpecName "kube-api-access-9t5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:23.401420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.401366 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07029a1f-f4a5-4f57-96a7-0574376c9822-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:23.401420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:23.401390 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5xj\" (UniqueName: \"kubernetes.io/projected/07029a1f-f4a5-4f57-96a7-0574376c9822-kube-api-access-9t5xj\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:24.137666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:24.137638 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" Nov 02 22:50:24.137831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:24.137666 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns" event={"ID":"07029a1f-f4a5-4f57-96a7-0574376c9822","Type":"ContainerDied","Data":"1281964289a2b56b43fb5959866d24d358a7c73243c7c69adbac64beed683e68"} Nov 02 22:50:24.137831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:24.137712 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1281964289a2b56b43fb5959866d24d358a7c73243c7c69adbac64beed683e68" Nov 02 22:50:25.943543 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:50:25.943511 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:50:36.991974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:36.991927 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw"] Nov 02 22:50:36.993348 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:36.993321 5859 cpu_manager.go:414] "containerMap: removing container" podUID="07029a1f-f4a5-4f57-96a7-0574376c9822" containerName="revoke-keys" Nov 02 22:50:36.993526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:36.993509 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07029a1f-f4a5-4f57-96a7-0574376c9822" containerName="revoke-keys" Nov 02 22:50:36.993769 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:36.993753 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="07029a1f-f4a5-4f57-96a7-0574376c9822" containerName="revoke-keys" Nov 02 22:50:37.002513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.002489 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.056972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.055866 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw"] Nov 02 22:50:37.113073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.113051 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:50:37.114099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.114079 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:50:37.114366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.114352 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:50:37.114589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.114575 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:50:37.165972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.158509 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.250132 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.250055 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpp5r\" (UniqueName: \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.396978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.396075 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.499429 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.499390 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.499706 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.499685 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpp5r\" (UniqueName: \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.499873 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.499856 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.502628 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.502575 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.547285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.547260 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.592004 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.581529 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpp5r\" (UniqueName: \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\") pod \"maas-api-revoke-sub-0b7e8edb-x6tqw\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:37.661140 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:37.661107 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:38.401400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:38.401039 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tt978 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:50:38.401400 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:38.401110 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tt978" podUID="e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:50:38.535968 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:38.535912 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw"] Nov 02 22:50:38.543364 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:50:38.543324 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160693ee_4f72_4fd9_b729_6c01a878c15e.slice/crio-449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe WatchSource:0}: Error finding container 449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe: Status 404 returned error can't find the container with id 449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe Nov 02 22:50:39.461894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:39.461865 5859 generic.go:356] "Generic (PLEG): container finished" podID="160693ee-4f72-4fd9-b729-6c01a878c15e" containerID="cc0ada6b6fec5fbfa26809ca689155de1401fad136edfd5c610cb8240537448c" exitCode=0 Nov 02 22:50:39.462253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:39.461966 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" event={"ID":"160693ee-4f72-4fd9-b729-6c01a878c15e","Type":"ContainerDied","Data":"cc0ada6b6fec5fbfa26809ca689155de1401fad136edfd5c610cb8240537448c"} Nov 02 22:50:39.462253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:39.462024 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" event={"ID":"160693ee-4f72-4fd9-b729-6c01a878c15e","Type":"ContainerStarted","Data":"449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe"} Nov 02 22:50:40.597652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.597628 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:40.686573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.686544 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\" (UniqueName: \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\") pod \"160693ee-4f72-4fd9-b729-6c01a878c15e\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " Nov 02 22:50:40.686722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.686584 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\") pod \"160693ee-4f72-4fd9-b729-6c01a878c15e\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " Nov 02 22:50:40.686722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.686650 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\") pod \"160693ee-4f72-4fd9-b729-6c01a878c15e\" (UID: \"160693ee-4f72-4fd9-b729-6c01a878c15e\") " Nov 02 22:50:40.687080 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.687051 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca" pod "160693ee-4f72-4fd9-b729-6c01a878c15e" (UID: "160693ee-4f72-4fd9-b729-6c01a878c15e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:40.688649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.688625 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r" pod "160693ee-4f72-4fd9-b729-6c01a878c15e" (UID: "160693ee-4f72-4fd9-b729-6c01a878c15e"). InnerVolumeSpecName "kube-api-access-lpp5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:40.688743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.688647 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca" pod "160693ee-4f72-4fd9-b729-6c01a878c15e" (UID: "160693ee-4f72-4fd9-b729-6c01a878c15e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:40.788454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.788403 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpp5r\" (UniqueName: \"kubernetes.io/projected/160693ee-4f72-4fd9-b729-6c01a878c15e-kube-api-access-lpp5r\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:40.788454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.788427 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:40.788454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:40.788437 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160693ee-4f72-4fd9-b729-6c01a878c15e-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:41.475338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:41.475306 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" Nov 02 22:50:41.475529 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:41.475380 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw" event={"ID":"160693ee-4f72-4fd9-b729-6c01a878c15e","Type":"ContainerDied","Data":"449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe"} Nov 02 22:50:41.475529 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:41.475446 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449680e9378246b8d70cfb2d4f46e0326479ec0c23588f197d24edbd036daebe" Nov 02 22:50:48.286766 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.286733 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv"] Nov 02 22:50:48.289316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.287190 5859 cpu_manager.go:414] "containerMap: removing container" podUID="160693ee-4f72-4fd9-b729-6c01a878c15e" containerName="revoke-keys" Nov 02 22:50:48.289316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.287211 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160693ee-4f72-4fd9-b729-6c01a878c15e" containerName="revoke-keys" Nov 02 22:50:48.289316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.287350 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="160693ee-4f72-4fd9-b729-6c01a878c15e" containerName="revoke-keys" Nov 02 22:50:48.293990 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.293975 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.344331 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.344305 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv"] Nov 02 22:50:48.361659 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.361636 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qv9f\" (UniqueName: \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.361788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.361702 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.361831 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.361794 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.388093 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.388067 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:50:48.388207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.388096 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:50:48.388207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.388097 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:50:48.389722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.389708 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:50:48.463249 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.463228 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qv9f\" (UniqueName: \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.463355 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.463293 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.463355 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.463340 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.463858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.463840 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.465552 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.465531 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.516518 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.516496 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qv9f\" (UniqueName: \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\") pod \"maas-api-revoke-sub-3d4156f1-5rmqv\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.614036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.614014 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:48.977295 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:48.977221 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv"] Nov 02 22:50:48.982523 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:50:48.982499 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89a8a8c_0ea0_4db9_9f39_34a68734e135.slice/crio-0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17 WatchSource:0}: Error finding container 0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17: Status 404 returned error can't find the container with id 0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17 Nov 02 22:50:49.520690 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:49.520662 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" event={"ID":"f89a8a8c-0ea0-4db9-9f39-34a68734e135","Type":"ContainerStarted","Data":"6cf983163e6867feca3abe526887733b361f5211ca15631e2c977cb27a182c59"} Nov 02 22:50:49.521045 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:49.520711 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" event={"ID":"f89a8a8c-0ea0-4db9-9f39-34a68734e135","Type":"ContainerStarted","Data":"0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17"} Nov 02 22:50:49.628732 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:49.628685 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" podStartSLOduration=1.6286732860000002 podStartE2EDuration="1.628673286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:50:49.626060039 +0000 UTC m=+6089.174309817" watchObservedRunningTime="2026-11-02 22:50:49.628673286 +0000 UTC m=+6089.176923066" Nov 02 22:50:50.527498 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:50.527466 5859 generic.go:356] "Generic (PLEG): container finished" podID="f89a8a8c-0ea0-4db9-9f39-34a68734e135" containerID="6cf983163e6867feca3abe526887733b361f5211ca15631e2c977cb27a182c59" exitCode=0 Nov 02 22:50:50.527902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:50.527557 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" event={"ID":"f89a8a8c-0ea0-4db9-9f39-34a68734e135","Type":"ContainerDied","Data":"6cf983163e6867feca3abe526887733b361f5211ca15631e2c977cb27a182c59"} Nov 02 22:50:51.668139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.668116 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:51.794780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.794717 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\" (UniqueName: \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\") pod \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " Nov 02 22:50:51.794780 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.794761 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\") pod \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " Nov 02 22:50:51.795025 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.794786 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\") pod \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\" (UID: \"f89a8a8c-0ea0-4db9-9f39-34a68734e135\") " Nov 02 22:50:51.795156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.795130 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca" pod "f89a8a8c-0ea0-4db9-9f39-34a68734e135" (UID: "f89a8a8c-0ea0-4db9-9f39-34a68734e135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:51.797287 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.797260 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca" pod "f89a8a8c-0ea0-4db9-9f39-34a68734e135" (UID: "f89a8a8c-0ea0-4db9-9f39-34a68734e135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:50:51.797736 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.797713 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f" pod "f89a8a8c-0ea0-4db9-9f39-34a68734e135" (UID: "f89a8a8c-0ea0-4db9-9f39-34a68734e135"). InnerVolumeSpecName "kube-api-access-7qv9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:51.895976 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.895931 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qv9f\" (UniqueName: \"kubernetes.io/projected/f89a8a8c-0ea0-4db9-9f39-34a68734e135-kube-api-access-7qv9f\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:51.896066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.895978 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:51.896066 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:51.895994 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89a8a8c-0ea0-4db9-9f39-34a68734e135-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:52.538860 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:52.538834 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" Nov 02 22:50:52.539047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:52.538871 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv" event={"ID":"f89a8a8c-0ea0-4db9-9f39-34a68734e135","Type":"ContainerDied","Data":"0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17"} Nov 02 22:50:52.539047 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:50:52.538914 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d15bcd09a162fbbc2011accbb5d6794416efda2f41075dc12eea24e3e300c17" Nov 02 22:50:55.944003 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:50:55.943967 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:51:14.556264 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.556225 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv"] Nov 02 22:51:14.602788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.602757 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rmqv"] Nov 02 22:51:14.674943 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.674905 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq"] Nov 02 22:51:14.675431 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.675410 5859 cpu_manager.go:414] "containerMap: removing container" podUID="f89a8a8c-0ea0-4db9-9f39-34a68734e135" containerName="revoke-keys" Nov 02 22:51:14.675530 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.675431 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89a8a8c-0ea0-4db9-9f39-34a68734e135" containerName="revoke-keys" Nov 02 22:51:14.675596 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.675583 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89a8a8c-0ea0-4db9-9f39-34a68734e135" containerName="revoke-keys" Nov 02 22:51:14.678931 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.678909 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.700691 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.700666 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgj6h\" (UniqueName: \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.700816 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.700728 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.700872 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.700859 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.729199 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.729174 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:51:14.729543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.729519 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:51:14.729543 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.729501 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:51:14.729727 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.729674 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:51:14.731739 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.731707 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq"] Nov 02 22:51:14.801934 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.801895 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgj6h\" (UniqueName: \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.802098 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.802020 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.802147 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.802133 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.802758 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.802734 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.804349 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.804327 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.855605 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.855577 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgj6h\" (UniqueName: \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\") pod \"maas-api-revoke-sub-3d4156f1-n4dkq\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:14.991154 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:14.991118 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:15.359974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:15.359913 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq"] Nov 02 22:51:15.366039 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:51:15.365998 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d4240f_15d9_4d1c_a969_2611cb35af05.slice/crio-831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5 WatchSource:0}: Error finding container 831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5: Status 404 returned error can't find the container with id 831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5 Nov 02 22:51:15.719044 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:15.719002 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" event={"ID":"23d4240f-15d9-4d1c-a969-2611cb35af05","Type":"ContainerStarted","Data":"6c13ad1e1711b22e54c7c3f44f905037755ba8b3b5674a6dd16b7b792cce875e"} Nov 02 22:51:15.719465 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:15.719061 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" event={"ID":"23d4240f-15d9-4d1c-a969-2611cb35af05","Type":"ContainerStarted","Data":"831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5"} Nov 02 22:51:15.824799 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:15.824743 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" podStartSLOduration=1.8247286950000001 podStartE2EDuration="1.824728695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:51:15.82398132 +0000 UTC m=+6115.372231102" watchObservedRunningTime="2026-11-02 22:51:15.824728695 +0000 UTC m=+6115.372978486" Nov 02 22:51:16.570041 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:16.570009 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89a8a8c-0ea0-4db9-9f39-34a68734e135" path="/var/lib/kubelet/pods/f89a8a8c-0ea0-4db9-9f39-34a68734e135/volumes" Nov 02 22:51:16.725539 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:16.725501 5859 generic.go:356] "Generic (PLEG): container finished" podID="23d4240f-15d9-4d1c-a969-2611cb35af05" containerID="6c13ad1e1711b22e54c7c3f44f905037755ba8b3b5674a6dd16b7b792cce875e" exitCode=0 Nov 02 22:51:16.725933 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:16.725543 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" event={"ID":"23d4240f-15d9-4d1c-a969-2611cb35af05","Type":"ContainerDied","Data":"6c13ad1e1711b22e54c7c3f44f905037755ba8b3b5674a6dd16b7b792cce875e"} Nov 02 22:51:17.871401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.871375 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:17.934891 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.934861 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\") pod \"23d4240f-15d9-4d1c-a969-2611cb35af05\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " Nov 02 22:51:17.935113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.934907 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\") pod \"23d4240f-15d9-4d1c-a969-2611cb35af05\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " Nov 02 22:51:17.935113 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.934934 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\" (UniqueName: \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\") pod \"23d4240f-15d9-4d1c-a969-2611cb35af05\" (UID: \"23d4240f-15d9-4d1c-a969-2611cb35af05\") " Nov 02 22:51:17.935335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.935311 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca" pod "23d4240f-15d9-4d1c-a969-2611cb35af05" (UID: "23d4240f-15d9-4d1c-a969-2611cb35af05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:51:17.935423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.935408 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:17.937119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.937085 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca" pod "23d4240f-15d9-4d1c-a969-2611cb35af05" (UID: "23d4240f-15d9-4d1c-a969-2611cb35af05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:17.937259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:17.937234 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h" pod "23d4240f-15d9-4d1c-a969-2611cb35af05" (UID: "23d4240f-15d9-4d1c-a969-2611cb35af05"). InnerVolumeSpecName "kube-api-access-xgj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:51:18.036607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:18.036520 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgj6h\" (UniqueName: \"kubernetes.io/projected/23d4240f-15d9-4d1c-a969-2611cb35af05-kube-api-access-xgj6h\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:18.036607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:18.036556 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d4240f-15d9-4d1c-a969-2611cb35af05-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:18.738456 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:18.738425 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" Nov 02 22:51:18.738611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:18.738461 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq" event={"ID":"23d4240f-15d9-4d1c-a969-2611cb35af05","Type":"ContainerDied","Data":"831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5"} Nov 02 22:51:18.738611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:18.738508 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831ee750a6af6829d14ad6b65579431cce86d8bac8a816908417c5e6db682fb5" Nov 02 22:51:22.808921 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:22.808886 5859 scope.go:122] "RemoveContainer" containerID="839f72150becee56d094b33c0ea3fc3dd7bb223951c03e370b4917d33cc8d437" Nov 02 22:51:25.943774 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:51:25.943739 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:51:55.945393 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:51:55.945354 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:51:59.641894 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.641853 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b"] Nov 02 22:51:59.642446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.642300 5859 cpu_manager.go:414] "containerMap: removing container" podUID="23d4240f-15d9-4d1c-a969-2611cb35af05" containerName="revoke-keys" Nov 02 22:51:59.642446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.642313 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d4240f-15d9-4d1c-a969-2611cb35af05" containerName="revoke-keys" Nov 02 22:51:59.642446 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.642408 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d4240f-15d9-4d1c-a969-2611cb35af05" containerName="revoke-keys" Nov 02 22:51:59.646001 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.645981 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.692453 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.692426 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rjc\" (UniqueName: \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.692607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.692473 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.692607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.692559 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.697082 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.697016 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:51:59.743738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.743697 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:51:59.743738 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.743697 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:51:59.744036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.743769 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:51:59.746917 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.746894 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b"] Nov 02 22:51:59.793574 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.793543 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.793721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.793585 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.793721 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.793680 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rjc\" (UniqueName: \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.794281 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.794257 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.796039 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.796020 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.846404 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.846339 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rjc\" (UniqueName: \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\") pod \"maas-api-revoke-sub-e2fc9c5b-xrx8b\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:51:59.958998 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:51:59.958910 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:52:00.325978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:00.325922 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b"] Nov 02 22:52:00.331438 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:52:00.331399 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec634eb_f817_4755_8a00_9b2def88bbd0.slice/crio-312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006 WatchSource:0}: Error finding container 312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006: Status 404 returned error can't find the container with id 312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006 Nov 02 22:52:01.044384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:01.044304 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" event={"ID":"9ec634eb-f817-4755-8a00-9b2def88bbd0","Type":"ContainerStarted","Data":"2d2335f6f77da282b7b12451ba5dedb2e4668cd25835c3eb83d446dedef53663"} Nov 02 22:52:01.044384 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:01.044365 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" event={"ID":"9ec634eb-f817-4755-8a00-9b2def88bbd0","Type":"ContainerStarted","Data":"312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006"} Nov 02 22:52:01.151124 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:01.151075 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" podStartSLOduration=2.151058397 podStartE2EDuration="2.151058397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:01.149486765 +0000 UTC m=+6160.697736528" watchObservedRunningTime="2026-11-02 22:52:01.151058397 +0000 UTC m=+6160.699308177" Nov 02 22:52:02.062148 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:02.062111 5859 generic.go:356] "Generic (PLEG): container finished" podID="9ec634eb-f817-4755-8a00-9b2def88bbd0" containerID="2d2335f6f77da282b7b12451ba5dedb2e4668cd25835c3eb83d446dedef53663" exitCode=0 Nov 02 22:52:02.062623 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:02.062196 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" event={"ID":"9ec634eb-f817-4755-8a00-9b2def88bbd0","Type":"ContainerDied","Data":"2d2335f6f77da282b7b12451ba5dedb2e4668cd25835c3eb83d446dedef53663"} Nov 02 22:52:03.203650 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.203623 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:52:03.329367 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.329277 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\" (UniqueName: \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\") pod \"9ec634eb-f817-4755-8a00-9b2def88bbd0\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " Nov 02 22:52:03.329519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.329448 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\") pod \"9ec634eb-f817-4755-8a00-9b2def88bbd0\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " Nov 02 22:52:03.329519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.329472 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\") pod \"9ec634eb-f817-4755-8a00-9b2def88bbd0\" (UID: \"9ec634eb-f817-4755-8a00-9b2def88bbd0\") " Nov 02 22:52:03.329847 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.329823 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca" pod "9ec634eb-f817-4755-8a00-9b2def88bbd0" (UID: "9ec634eb-f817-4755-8a00-9b2def88bbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:52:03.331422 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.331399 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc" pod "9ec634eb-f817-4755-8a00-9b2def88bbd0" (UID: "9ec634eb-f817-4755-8a00-9b2def88bbd0"). InnerVolumeSpecName "kube-api-access-f7rjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:52:03.331589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.331557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca" pod "9ec634eb-f817-4755-8a00-9b2def88bbd0" (UID: "9ec634eb-f817-4755-8a00-9b2def88bbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:03.431234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.431196 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7rjc\" (UniqueName: \"kubernetes.io/projected/9ec634eb-f817-4755-8a00-9b2def88bbd0-kube-api-access-f7rjc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:03.431234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.431227 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:03.431234 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:03.431239 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec634eb-f817-4755-8a00-9b2def88bbd0-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:04.075161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:04.075132 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" Nov 02 22:52:04.075316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:04.075160 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b" event={"ID":"9ec634eb-f817-4755-8a00-9b2def88bbd0","Type":"ContainerDied","Data":"312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006"} Nov 02 22:52:04.075316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:04.075205 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312312a3330e7ad5fd0c21a1e4eb3cfc68d42e7337c8d752562f314b74d1b006" Nov 02 22:52:09.044406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:09.044371 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c"] Nov 02 22:52:09.090825 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:09.090799 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dxb9c"] Nov 02 22:52:10.426671 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.426639 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p"] Nov 02 22:52:10.427139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.427116 5859 cpu_manager.go:414] "containerMap: removing container" podUID="9ec634eb-f817-4755-8a00-9b2def88bbd0" containerName="revoke-keys" Nov 02 22:52:10.427139 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.427132 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec634eb-f817-4755-8a00-9b2def88bbd0" containerName="revoke-keys" Nov 02 22:52:10.427316 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.427196 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec634eb-f817-4755-8a00-9b2def88bbd0" containerName="revoke-keys" Nov 02 22:52:10.430557 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.430541 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.481829 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.481807 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:52:10.505837 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.505807 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.505992 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.505972 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.506056 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.506016 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg22w\" (UniqueName: \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.527944 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.527921 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:52:10.528112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.528027 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:52:10.528112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.528086 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:52:10.530565 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.530542 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p"] Nov 02 22:52:10.570816 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.570790 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3824e39b-2395-4ba2-b81b-67cbfbfb6734" path="/var/lib/kubelet/pods/3824e39b-2395-4ba2-b81b-67cbfbfb6734/volumes" Nov 02 22:52:10.606822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.606791 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.607013 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.606830 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg22w\" (UniqueName: \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.607013 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.606885 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.607516 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.607494 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.609181 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.609165 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.659647 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.659620 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg22w\" (UniqueName: \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\") pod \"maas-api-revoke-sub-5a29efb3-pjh2p\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:10.743149 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:10.743077 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:11.110018 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:11.109986 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p"] Nov 02 22:52:11.117143 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:52:11.117111 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce903ad_a6f4_4231_a479_fd4761121c12.slice/crio-4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c WatchSource:0}: Error finding container 4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c: Status 404 returned error can't find the container with id 4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c Nov 02 22:52:12.122911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:12.122878 5859 generic.go:356] "Generic (PLEG): container finished" podID="cce903ad-a6f4-4231-a479-fd4761121c12" containerID="ba7fa966511c11c0674af0fda5e76d93e10c379130862b073d4adccf1648d92e" exitCode=0 Nov 02 22:52:12.123387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:12.122968 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" event={"ID":"cce903ad-a6f4-4231-a479-fd4761121c12","Type":"ContainerDied","Data":"ba7fa966511c11c0674af0fda5e76d93e10c379130862b073d4adccf1648d92e"} Nov 02 22:52:12.123387 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:12.123015 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" event={"ID":"cce903ad-a6f4-4231-a479-fd4761121c12","Type":"ContainerStarted","Data":"4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c"} Nov 02 22:52:13.368866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.368826 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:13.467600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.467481 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\" (UniqueName: \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\") pod \"cce903ad-a6f4-4231-a479-fd4761121c12\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " Nov 02 22:52:13.468072 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.468045 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\") pod \"cce903ad-a6f4-4231-a479-fd4761121c12\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " Nov 02 22:52:13.468649 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.468622 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca" pod "cce903ad-a6f4-4231-a479-fd4761121c12" (UID: "cce903ad-a6f4-4231-a479-fd4761121c12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:52:13.468763 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.468748 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\") pod \"cce903ad-a6f4-4231-a479-fd4761121c12\" (UID: \"cce903ad-a6f4-4231-a479-fd4761121c12\") " Nov 02 22:52:13.469259 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.469236 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:13.470526 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.470493 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w" pod "cce903ad-a6f4-4231-a479-fd4761121c12" (UID: "cce903ad-a6f4-4231-a479-fd4761121c12"). InnerVolumeSpecName "kube-api-access-cg22w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:52:13.470638 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.470582 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca" pod "cce903ad-a6f4-4231-a479-fd4761121c12" (UID: "cce903ad-a6f4-4231-a479-fd4761121c12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:13.570918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.570861 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg22w\" (UniqueName: \"kubernetes.io/projected/cce903ad-a6f4-4231-a479-fd4761121c12-kube-api-access-cg22w\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:13.570918 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:13.570912 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce903ad-a6f4-4231-a479-fd4761121c12-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:14.197995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:14.194688 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" event={"ID":"cce903ad-a6f4-4231-a479-fd4761121c12","Type":"ContainerDied","Data":"4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c"} Nov 02 22:52:14.197995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:14.194753 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b22417da5a3dd3be333e742abc1b4a3937da2cb4f724c62b9022e1b4bf72a6c" Nov 02 22:52:14.197995 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:14.194831 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p" Nov 02 22:52:25.945074 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:52:25.945021 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:52:29.033327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.033292 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq"] Nov 02 22:52:29.037624 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.037597 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n4dkq"] Nov 02 22:52:29.149563 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.149524 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8"] Nov 02 22:52:29.149991 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.149978 5859 cpu_manager.go:414] "containerMap: removing container" podUID="cce903ad-a6f4-4231-a479-fd4761121c12" containerName="revoke-keys" Nov 02 22:52:29.150050 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.149992 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce903ad-a6f4-4231-a479-fd4761121c12" containerName="revoke-keys" Nov 02 22:52:29.150099 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.150070 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce903ad-a6f4-4231-a479-fd4761121c12" containerName="revoke-keys" Nov 02 22:52:29.153612 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.153592 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.204689 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.204651 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:52:29.204928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.204729 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:52:29.204928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.204744 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:52:29.204928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.204794 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:52:29.208219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.208177 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8"] Nov 02 22:52:29.255677 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.255652 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.255808 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.255778 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.255851 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.255825 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59t5\" (UniqueName: \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.357283 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.357247 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59t5\" (UniqueName: \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.357454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.357297 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.357454 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.357387 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.357989 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.357965 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.359660 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.359638 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.410211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.410191 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59t5\" (UniqueName: \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\") pod \"maas-api-revoke-sub-3d4156f1-stpf8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.466212 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.466188 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:29.830564 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:29.830536 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8"] Nov 02 22:52:29.835836 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:52:29.835811 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e15d40_4c6c_41e4_982f_7187680139e8.slice/crio-5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e WatchSource:0}: Error finding container 5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e: Status 404 returned error can't find the container with id 5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e Nov 02 22:52:30.304335 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:30.304302 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" event={"ID":"60e15d40-4c6c-41e4-982f-7187680139e8","Type":"ContainerStarted","Data":"9393db7b0ed89cafbc001cd1b3624ac3d09cb527501c60289af72321f27a090e"} Nov 02 22:52:30.304744 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:30.304359 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" event={"ID":"60e15d40-4c6c-41e4-982f-7187680139e8","Type":"ContainerStarted","Data":"5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e"} Nov 02 22:52:30.414474 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:30.414397 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" podStartSLOduration=1.414384091 podStartE2EDuration="1.414384091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:30.412352465 +0000 UTC m=+6189.960602262" watchObservedRunningTime="2026-11-02 22:52:30.414384091 +0000 UTC m=+6189.962633871" Nov 02 22:52:30.569872 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:30.569841 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d4240f-15d9-4d1c-a969-2611cb35af05" path="/var/lib/kubelet/pods/23d4240f-15d9-4d1c-a969-2611cb35af05/volumes" Nov 02 22:52:31.311793 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:31.311760 5859 generic.go:356] "Generic (PLEG): container finished" podID="60e15d40-4c6c-41e4-982f-7187680139e8" containerID="9393db7b0ed89cafbc001cd1b3624ac3d09cb527501c60289af72321f27a090e" exitCode=0 Nov 02 22:52:31.312241 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:31.311852 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" event={"ID":"60e15d40-4c6c-41e4-982f-7187680139e8","Type":"ContainerDied","Data":"9393db7b0ed89cafbc001cd1b3624ac3d09cb527501c60289af72321f27a090e"} Nov 02 22:52:32.448770 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.448742 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:32.486560 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.486537 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\") pod \"60e15d40-4c6c-41e4-982f-7187680139e8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " Nov 02 22:52:32.486706 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.486640 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\" (UniqueName: \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\") pod \"60e15d40-4c6c-41e4-982f-7187680139e8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " Nov 02 22:52:32.486774 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.486725 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\") pod \"60e15d40-4c6c-41e4-982f-7187680139e8\" (UID: \"60e15d40-4c6c-41e4-982f-7187680139e8\") " Nov 02 22:52:32.487104 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.487079 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca" pod "60e15d40-4c6c-41e4-982f-7187680139e8" (UID: "60e15d40-4c6c-41e4-982f-7187680139e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:52:32.488577 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.488557 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca" pod "60e15d40-4c6c-41e4-982f-7187680139e8" (UID: "60e15d40-4c6c-41e4-982f-7187680139e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:32.488662 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.488647 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5" pod "60e15d40-4c6c-41e4-982f-7187680139e8" (UID: "60e15d40-4c6c-41e4-982f-7187680139e8"). InnerVolumeSpecName "kube-api-access-q59t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:52:32.587483 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.587457 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59t5\" (UniqueName: \"kubernetes.io/projected/60e15d40-4c6c-41e4-982f-7187680139e8-kube-api-access-q59t5\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:32.587483 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.587482 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:32.587637 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:32.587492 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e15d40-4c6c-41e4-982f-7187680139e8-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:33.325615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:33.325586 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" Nov 02 22:52:33.325796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:33.325588 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8" event={"ID":"60e15d40-4c6c-41e4-982f-7187680139e8","Type":"ContainerDied","Data":"5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e"} Nov 02 22:52:33.325796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:33.325705 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5764910bf16900d5b27dd1dea89fb295bc1a4ada800112327b1b7c251d16e73e" Nov 02 22:52:43.057432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:43.057377 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx"] Nov 02 22:52:43.100489 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:43.100447 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gw4lx"] Nov 02 22:52:44.569761 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:44.569714 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d253cdd-1010-4918-8be2-4fc1336e3674" path="/var/lib/kubelet/pods/2d253cdd-1010-4918-8be2-4fc1336e3674/volumes" Nov 02 22:52:55.945048 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:52:55.945006 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:52:56.381928 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.381889 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:52:56.382482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.382457 5859 cpu_manager.go:414] "containerMap: removing container" podUID="60e15d40-4c6c-41e4-982f-7187680139e8" containerName="revoke-keys" Nov 02 22:52:56.382482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.382480 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e15d40-4c6c-41e4-982f-7187680139e8" containerName="revoke-keys" Nov 02 22:52:56.382666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.382606 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e15d40-4c6c-41e4-982f-7187680139e8" containerName="revoke-keys" Nov 02 22:52:56.386073 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.386054 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.440745 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.440715 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:52:56.484523 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.484488 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-90aa9a\"" Nov 02 22:52:56.484733 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.484547 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1ae3c580-dockercfg-db7k5\"" Nov 02 22:52:56.497464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.497164 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.497464 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.497385 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9492c\" (UniqueName: \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.555230 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.555190 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:52:56.559033 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.559012 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.598242 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.598220 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.598358 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.598340 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9492c\" (UniqueName: \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.598923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.598903 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.612467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.612447 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:52:56.651436 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.651374 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9492c\" (UniqueName: \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\") pod \"payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.698974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.698918 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdkc\" (UniqueName: \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.699146 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.698997 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.699636 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.699610 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:56.799997 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.799924 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdkc\" (UniqueName: \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.800156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.800057 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.800925 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.800906 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.852830 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.852802 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdkc\" (UniqueName: \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\") pod \"payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:56.872596 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:56.872568 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:57.072743 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.072703 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:52:57.077760 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:52:57.077729 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba85d1_b49c_491c_83d3_b74d4e82e515.slice/crio-ad2fbe2e03881a7b09d0f7079119cf5d09e75a678f6e2cc020fd975d69c7e1ea WatchSource:0}: Error finding container ad2fbe2e03881a7b09d0f7079119cf5d09e75a678f6e2cc020fd975d69c7e1ea: Status 404 returned error can't find the container with id ad2fbe2e03881a7b09d0f7079119cf5d09e75a678f6e2cc020fd975d69c7e1ea Nov 02 22:52:57.239581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.239491 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:52:57.244755 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:52:57.244728 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f494f6_21a1_4dd5_9546_e2b0f913d5d4.slice/crio-275d1bd94186ff65d7a505d21791485f679f939776b07c11c0f60c903f3072eb WatchSource:0}: Error finding container 275d1bd94186ff65d7a505d21791485f679f939776b07c11c0f60c903f3072eb: Status 404 returned error can't find the container with id 275d1bd94186ff65d7a505d21791485f679f939776b07c11c0f60c903f3072eb Nov 02 22:52:57.490467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.490375 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" event={"ID":"57f494f6-21a1-4dd5-9546-e2b0f913d5d4","Type":"ContainerStarted","Data":"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796"} Nov 02 22:52:57.490467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.490437 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" event={"ID":"57f494f6-21a1-4dd5-9546-e2b0f913d5d4","Type":"ContainerStarted","Data":"275d1bd94186ff65d7a505d21791485f679f939776b07c11c0f60c903f3072eb"} Nov 02 22:52:57.490467 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.490464 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:52:57.492090 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.492066 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" event={"ID":"b3ba85d1-b49c-491c-83d3-b74d4e82e515","Type":"ContainerStarted","Data":"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6"} Nov 02 22:52:57.492228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.492104 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" event={"ID":"b3ba85d1-b49c-491c-83d3-b74d4e82e515","Type":"ContainerStarted","Data":"ad2fbe2e03881a7b09d0f7079119cf5d09e75a678f6e2cc020fd975d69c7e1ea"} Nov 02 22:52:57.492228 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.492124 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:52:57.598789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.598740 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" podStartSLOduration=1.598726434 podStartE2EDuration="1.598726434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:57.595689927 +0000 UTC m=+6217.143939701" watchObservedRunningTime="2026-11-02 22:52:57.598726434 +0000 UTC m=+6217.146976214" Nov 02 22:52:57.703801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:57.703750 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" podStartSLOduration=1.7037360179999999 podStartE2EDuration="1.703736018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:57.700888983 +0000 UTC m=+6217.249138778" watchObservedRunningTime="2026-11-02 22:52:57.703736018 +0000 UTC m=+6217.251985798" Nov 02 22:52:59.257481 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.257447 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:52:59.261795 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.261774 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.346866 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.346827 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrwx\" (UniqueName: \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.347077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.346876 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.347077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.347031 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.347178 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.347121 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.358624 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.358591 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-90aa9a-serving-cert\"" Nov 02 22:52:59.358788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.358630 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:52:59.358788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.358673 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:52:59.358788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.358710 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:52:59.358788 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.358781 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 22:52:59.361225 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.361202 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:52:59.448648 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.448611 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.448840 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.448704 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.448840 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:52:59.448778 5859 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-serving-cert: secret "maas-api-e2e-worker-single-main-90aa9a-serving-cert" not found Nov 02 22:52:59.448840 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.448794 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.449036 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:52:59.448858 5859 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls podName:c268e401-6d8e-4029-bd61-416cf57b8c79 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:59.948836883 +0000 UTC m=+6219.497086648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls") pod "maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" (UID: "c268e401-6d8e-4029-bd61-416cf57b8c79") : secret "maas-api-e2e-worker-single-main-90aa9a-serving-cert" not found Nov 02 22:52:59.449036 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.448890 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrwx\" (UniqueName: \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.449493 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.449472 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.451276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.451252 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.503621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.503584 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrwx\" (UniqueName: \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.953207 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.953172 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:52:59.955682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:52:59.955650 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:53:00.175683 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:00.175646 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:53:00.545814 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:00.545735 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:53:00.551131 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:53:00.551108 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc268e401_6d8e_4029_bd61_416cf57b8c79.slice/crio-e7b9c30d421bf3422c9da650713a0ac808736cafda529b2d10b9643cdc6aa72b WatchSource:0}: Error finding container e7b9c30d421bf3422c9da650713a0ac808736cafda529b2d10b9643cdc6aa72b: Status 404 returned error can't find the container with id e7b9c30d421bf3422c9da650713a0ac808736cafda529b2d10b9643cdc6aa72b Nov 02 22:53:00.552536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:00.552519 5859 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:53:01.516942 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:01.516910 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" event={"ID":"c268e401-6d8e-4029-bd61-416cf57b8c79","Type":"ContainerStarted","Data":"e7b9c30d421bf3422c9da650713a0ac808736cafda529b2d10b9643cdc6aa72b"} Nov 02 22:53:02.525260 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:02.525219 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" event={"ID":"c268e401-6d8e-4029-bd61-416cf57b8c79","Type":"ContainerStarted","Data":"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395"} Nov 02 22:53:02.525723 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:02.525353 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:53:02.633759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:02.633717 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" podStartSLOduration=2.638719132 podStartE2EDuration="3.633705292s" totalImagesPullingTime="994.98616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:02.631306861 +0000 UTC m=+6222.179556653" watchObservedRunningTime="2026-11-02 22:53:02.633705292 +0000 UTC m=+6222.181955073" Nov 02 22:53:08.042560 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.042523 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p"] Nov 02 22:53:08.088941 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.088910 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4jj4p"] Nov 02 22:53:08.500058 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.500034 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:53:08.500257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.500105 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:53:08.536686 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.536664 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:53:08.580669 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:08.580632 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929f3fa9-01ed-4db4-a79e-82b86ce32c61" path="/var/lib/kubelet/pods/929f3fa9-01ed-4db4-a79e-82b86ce32c61/volumes" Nov 02 22:53:20.043482 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:20.043451 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6"] Nov 02 22:53:20.089799 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:20.089770 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4vqf6"] Nov 02 22:53:20.569826 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:20.569791 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f81c82f-d32f-4856-ae3c-6d62cdce1f35" path="/var/lib/kubelet/pods/3f81c82f-d32f-4856-ae3c-6d62cdce1f35/volumes" Nov 02 22:53:22.840575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:22.840546 5859 scope.go:122] "RemoveContainer" containerID="4b2a10b29465bd43d6ca042b61d8f23e1b439669acb231b5aebeb1b37d8a8a2a" Nov 02 22:53:25.944077 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:53:25.944042 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:53:55.045569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:55.045537 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4"] Nov 02 22:53:55.091800 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:55.091777 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-chnh4"] Nov 02 22:53:55.944382 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:53:55.944345 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:53:56.570102 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:53:56.570072 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b999379d-181e-4d2b-acbe-8b1c9791c854" path="/var/lib/kubelet/pods/b999379d-181e-4d2b-acbe-8b1c9791c854/volumes" Nov 02 22:54:06.042126 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:06.042094 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz"] Nov 02 22:54:06.088576 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:06.088541 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jnqtz"] Nov 02 22:54:06.569552 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:06.569518 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027621f8-0c60-4c9a-b98c-1896046a7bad" path="/var/lib/kubelet/pods/027621f8-0c60-4c9a-b98c-1896046a7bad/volumes" Nov 02 22:54:22.866324 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:22.866289 5859 scope.go:122] "RemoveContainer" containerID="3057a9c5cd3a43241fef712ec6e29b998e268a7c5da5939f174eb08e07b4fdae" Nov 02 22:54:22.878659 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:22.878629 5859 scope.go:122] "RemoveContainer" containerID="4fdc213ffdf1a2d1047e4cef3ee05f7f279c1269ed0b7767de0b46785faf2f1b" Nov 02 22:54:22.888699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:22.888675 5859 scope.go:122] "RemoveContainer" containerID="46da49a3d006f6c9f25e27e4c5acaf5f8ef82d5cdef0845d7331034c8eacaefd" Nov 02 22:54:25.944659 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:54:25.944618 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:54:26.076759 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:26.076726 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:54:26.174401 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:26.174369 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:54:27.274491 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:27.274459 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:54:27.274893 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:27.274662 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" podUID="c268e401-6d8e-4029-bd61-416cf57b8c79" containerName="maas-api" containerID="cri-o://97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395" gracePeriod=30 Nov 02 22:54:28.526652 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.526621 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:54:28.578007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.577977 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") pod \"c268e401-6d8e-4029-bd61-416cf57b8c79\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " Nov 02 22:54:28.578138 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.578045 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\") pod \"c268e401-6d8e-4029-bd61-416cf57b8c79\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " Nov 02 22:54:28.578182 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.578144 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\") pod \"c268e401-6d8e-4029-bd61-416cf57b8c79\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " Nov 02 22:54:28.578223 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.578189 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\" (UniqueName: \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\") pod \"c268e401-6d8e-4029-bd61-416cf57b8c79\" (UID: \"c268e401-6d8e-4029-bd61-416cf57b8c79\") " Nov 02 22:54:28.578629 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.578517 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca" pod "c268e401-6d8e-4029-bd61-416cf57b8c79" (UID: "c268e401-6d8e-4029-bd61-416cf57b8c79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:28.578629 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.578608 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c268e401-6d8e-4029-bd61-416cf57b8c79-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:28.580037 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.580016 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls" pod "c268e401-6d8e-4029-bd61-416cf57b8c79" (UID: "c268e401-6d8e-4029-bd61-416cf57b8c79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:28.580215 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.580180 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx" pod "c268e401-6d8e-4029-bd61-416cf57b8c79" (UID: "c268e401-6d8e-4029-bd61-416cf57b8c79"). InnerVolumeSpecName "kube-api-access-dkrwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:28.580285 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.580266 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs" pod "c268e401-6d8e-4029-bd61-416cf57b8c79" (UID: "c268e401-6d8e-4029-bd61-416cf57b8c79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:28.679511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.679444 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrwx\" (UniqueName: \"kubernetes.io/projected/c268e401-6d8e-4029-bd61-416cf57b8c79-kube-api-access-dkrwx\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:28.679511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.679473 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:28.679511 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:28.679483 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c268e401-6d8e-4029-bd61-416cf57b8c79-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:29.046594 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.046518 5859 generic.go:356] "Generic (PLEG): container finished" podID="c268e401-6d8e-4029-bd61-416cf57b8c79" containerID="97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395" exitCode=0 Nov 02 22:54:29.046594 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.046564 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" Nov 02 22:54:29.046776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.046610 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" event={"ID":"c268e401-6d8e-4029-bd61-416cf57b8c79","Type":"ContainerDied","Data":"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395"} Nov 02 22:54:29.046776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.046661 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq" event={"ID":"c268e401-6d8e-4029-bd61-416cf57b8c79","Type":"ContainerDied","Data":"e7b9c30d421bf3422c9da650713a0ac808736cafda529b2d10b9643cdc6aa72b"} Nov 02 22:54:29.046776 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.046682 5859 scope.go:122] "RemoveContainer" containerID="97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395" Nov 02 22:54:29.056076 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.056060 5859 scope.go:122] "RemoveContainer" containerID="97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395" Nov 02 22:54:29.056362 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:54:29.056344 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395\": container with ID starting with 97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395 not found: ID does not exist" containerID="97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395" Nov 02 22:54:29.056431 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.056367 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395"} err="failed to get container status \"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395\": rpc error: code = NotFound desc = could not find container \"97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395\": container with ID starting with 97eb678c947d45d54df9502f04634add9dbc80d30585cdfe6361341db05cc395 not found: ID does not exist" Nov 02 22:54:29.202277 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.202253 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:54:29.248296 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:29.248268 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-90aa9a-78d69865b5-l82wq"] Nov 02 22:54:30.576672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:30.576640 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c268e401-6d8e-4029-bd61-416cf57b8c79" path="/var/lib/kubelet/pods/c268e401-6d8e-4029-bd61-416cf57b8c79/volumes" Nov 02 22:54:31.077710 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:31.077675 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" containerID="cri-o://4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796" gracePeriod=25 Nov 02 22:54:31.175445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:31.175386 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" containerID="cri-o://fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6" gracePeriod=25 Nov 02 22:54:38.498902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:38.498824 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:54:38.498902 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:38.498835 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:54:40.044434 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:40.044396 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff"] Nov 02 22:54:40.090449 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:40.090426 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vmrff"] Nov 02 22:54:40.569665 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:40.569632 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10bc770-124c-4384-b084-ebee7eed1de0" path="/var/lib/kubelet/pods/a10bc770-124c-4384-b084-ebee7eed1de0/volumes" Nov 02 22:54:48.498895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:48.498858 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:54:48.498895 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:48.498878 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:54:51.326114 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.326086 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:54:51.420985 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.420939 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:54:51.480477 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.480447 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\" (UniqueName: \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\") pod \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " Nov 02 22:54:51.480617 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.480534 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\") pod \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\" (UID: \"57f494f6-21a1-4dd5-9546-e2b0f913d5d4\") " Nov 02 22:54:51.480901 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.480878 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume" pod "57f494f6-21a1-4dd5-9546-e2b0f913d5d4" (UID: "57f494f6-21a1-4dd5-9546-e2b0f913d5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:51.481024 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.481010 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:51.482600 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.482576 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc" pod "57f494f6-21a1-4dd5-9546-e2b0f913d5d4" (UID: "57f494f6-21a1-4dd5-9546-e2b0f913d5d4"). InnerVolumeSpecName "kube-api-access-2zdkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:51.581581 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.581554 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\" (UniqueName: \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\") pod \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " Nov 02 22:54:51.581731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.581696 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\") pod \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\" (UID: \"b3ba85d1-b49c-491c-83d3-b74d4e82e515\") " Nov 02 22:54:51.582031 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.582013 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zdkc\" (UniqueName: \"kubernetes.io/projected/57f494f6-21a1-4dd5-9546-e2b0f913d5d4-kube-api-access-2zdkc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:51.582117 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.582091 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume" pod "b3ba85d1-b49c-491c-83d3-b74d4e82e515" (UID: "b3ba85d1-b49c-491c-83d3-b74d4e82e515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:54:51.583561 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.583537 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c" pod "b3ba85d1-b49c-491c-83d3-b74d4e82e515" (UID: "b3ba85d1-b49c-491c-83d3-b74d4e82e515"). InnerVolumeSpecName "kube-api-access-9492c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:51.682923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.682876 5859 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ba85d1-b49c-491c-83d3-b74d4e82e515-plugins-config-volume\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:51.682923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:51.682896 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9492c\" (UniqueName: \"kubernetes.io/projected/b3ba85d1-b49c-491c-83d3-b74d4e82e515-kube-api-access-9492c\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:52.180359 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.180328 5859 generic.go:356] "Generic (PLEG): container finished" podID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerID="fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6" exitCode=0 Nov 02 22:54:52.180519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.180386 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" Nov 02 22:54:52.180519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.180414 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" event={"ID":"b3ba85d1-b49c-491c-83d3-b74d4e82e515","Type":"ContainerDied","Data":"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6"} Nov 02 22:54:52.180519 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.180502 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv" event={"ID":"b3ba85d1-b49c-491c-83d3-b74d4e82e515","Type":"ContainerDied","Data":"ad2fbe2e03881a7b09d0f7079119cf5d09e75a678f6e2cc020fd975d69c7e1ea"} Nov 02 22:54:52.180672 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.180523 5859 scope.go:122] "RemoveContainer" containerID="fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6" Nov 02 22:54:52.182007 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.181986 5859 generic.go:356] "Generic (PLEG): container finished" podID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerID="4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796" exitCode=0 Nov 02 22:54:52.182123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.182054 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" Nov 02 22:54:52.182123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.182078 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" event={"ID":"57f494f6-21a1-4dd5-9546-e2b0f913d5d4","Type":"ContainerDied","Data":"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796"} Nov 02 22:54:52.182243 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.182154 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k" event={"ID":"57f494f6-21a1-4dd5-9546-e2b0f913d5d4","Type":"ContainerDied","Data":"275d1bd94186ff65d7a505d21791485f679f939776b07c11c0f60c903f3072eb"} Nov 02 22:54:52.192732 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.192713 5859 scope.go:122] "RemoveContainer" containerID="fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6" Nov 02 22:54:52.193316 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:54:52.193271 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6\": container with ID starting with fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6 not found: ID does not exist" containerID="fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6" Nov 02 22:54:52.193504 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.193472 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6"} err="failed to get container status \"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6\": rpc error: code = NotFound desc = could not find container \"fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6\": container with ID starting with fa0bce001b2d760fbd78e76b0ca5e0a98965a43829975a0ab88878478154eae6 not found: ID does not exist" Nov 02 22:54:52.193611 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.193505 5859 scope.go:122] "RemoveContainer" containerID="4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796" Nov 02 22:54:52.205181 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.205164 5859 scope.go:122] "RemoveContainer" containerID="4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796" Nov 02 22:54:52.205513 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:54:52.205488 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796\": container with ID starting with 4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796 not found: ID does not exist" containerID="4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796" Nov 02 22:54:52.205565 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.205524 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796"} err="failed to get container status \"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796\": rpc error: code = NotFound desc = could not find container \"4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796\": container with ID starting with 4e1e1aa451efb7dceebbbbdffd6d74f8990ccb0b4bfb30e873f4931d80723796 not found: ID does not exist" Nov 02 22:54:52.332897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.332872 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:54:52.378878 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.378812 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-90aa9a-7fbff6jxzv"] Nov 02 22:54:52.436339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.436313 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:54:52.440595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.440569 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-90aa9a-6b4578bb9drm9k"] Nov 02 22:54:52.569513 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.569488 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" path="/var/lib/kubelet/pods/57f494f6-21a1-4dd5-9546-e2b0f913d5d4/volumes" Nov 02 22:54:52.569861 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:54:52.569848 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" path="/var/lib/kubelet/pods/b3ba85d1-b49c-491c-83d3-b74d4e82e515/volumes" Nov 02 22:54:55.945532 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:54:55.945498 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:55:03.043443 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:03.043400 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz"] Nov 02 22:55:03.088943 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:03.088900 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-x54mz"] Nov 02 22:55:04.569863 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:04.569828 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e5d064-bb03-4ff2-9114-683434a7a88c" path="/var/lib/kubelet/pods/78e5d064-bb03-4ff2-9114-683434a7a88c/volumes" Nov 02 22:55:05.963573 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:05.963534 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8"] Nov 02 22:55:06.009911 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.009881 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-stpf8"] Nov 02 22:55:06.116151 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116110 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt"] Nov 02 22:55:06.116647 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116628 5859 cpu_manager.go:414] "containerMap: removing container" podUID="c268e401-6d8e-4029-bd61-416cf57b8c79" containerName="maas-api" Nov 02 22:55:06.116748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116649 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c268e401-6d8e-4029-bd61-416cf57b8c79" containerName="maas-api" Nov 02 22:55:06.116748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116665 5859 cpu_manager.go:414] "containerMap: removing container" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" Nov 02 22:55:06.116748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116673 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" Nov 02 22:55:06.116748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116718 5859 cpu_manager.go:414] "containerMap: removing container" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" Nov 02 22:55:06.116748 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116726 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" Nov 02 22:55:06.117048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116833 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ba85d1-b49c-491c-83d3-b74d4e82e515" containerName="payload-pre-processing" Nov 02 22:55:06.117048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116856 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="c268e401-6d8e-4029-bd61-416cf57b8c79" containerName="maas-api" Nov 02 22:55:06.117048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.116864 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f494f6-21a1-4dd5-9546-e2b0f913d5d4" containerName="payload-processing" Nov 02 22:55:06.121666 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.121644 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.170908 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.170886 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:55:06.171075 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.171051 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:55:06.171153 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.171138 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:55:06.171219 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.171174 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:55:06.173659 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.173640 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt"] Nov 02 22:55:06.216327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.216273 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.216327 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.216315 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95sb\" (UniqueName: \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.216472 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.216429 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.317842 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.317809 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.318043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.317860 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.318043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.317892 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95sb\" (UniqueName: \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.318462 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.318442 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.320279 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.320259 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.370615 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.370584 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95sb\" (UniqueName: \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\") pod \"maas-api-revoke-sub-3d4156f1-t6tdt\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.434624 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.434596 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:06.570019 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.569992 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e15d40-4c6c-41e4-982f-7187680139e8" path="/var/lib/kubelet/pods/60e15d40-4c6c-41e4-982f-7187680139e8/volumes" Nov 02 22:55:06.799874 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:06.799805 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt"] Nov 02 22:55:06.806488 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:55:06.806451 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9321483f_6ff9_4f37_a4c6_87a1cc63f455.slice/crio-36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4 WatchSource:0}: Error finding container 36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4: Status 404 returned error can't find the container with id 36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4 Nov 02 22:55:07.274747 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:07.274712 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" event={"ID":"9321483f-6ff9-4f37-a4c6-87a1cc63f455","Type":"ContainerStarted","Data":"b8df5e693cf4bd95b05ec21d8e5c39cf768fbeb9f88db1c975640686c63b6846"} Nov 02 22:55:07.275103 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:07.274775 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" event={"ID":"9321483f-6ff9-4f37-a4c6-87a1cc63f455","Type":"ContainerStarted","Data":"36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4"} Nov 02 22:55:07.382813 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:07.382768 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" podStartSLOduration=1.382753961 podStartE2EDuration="1.382753961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:55:07.382470195 +0000 UTC m=+6346.930719987" watchObservedRunningTime="2026-11-02 22:55:07.382753961 +0000 UTC m=+6346.931003742" Nov 02 22:55:08.282595 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:08.282560 5859 generic.go:356] "Generic (PLEG): container finished" podID="9321483f-6ff9-4f37-a4c6-87a1cc63f455" containerID="b8df5e693cf4bd95b05ec21d8e5c39cf768fbeb9f88db1c975640686c63b6846" exitCode=0 Nov 02 22:55:08.282975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:08.282641 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" event={"ID":"9321483f-6ff9-4f37-a4c6-87a1cc63f455","Type":"ContainerDied","Data":"b8df5e693cf4bd95b05ec21d8e5c39cf768fbeb9f88db1c975640686c63b6846"} Nov 02 22:55:09.419771 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.419743 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:09.447858 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.447834 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\") pod \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " Nov 02 22:55:09.448090 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.448069 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\") pod \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " Nov 02 22:55:09.448173 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.448115 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\" (UniqueName: \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\") pod \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\" (UID: \"9321483f-6ff9-4f37-a4c6-87a1cc63f455\") " Nov 02 22:55:09.448253 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.448222 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca" pod "9321483f-6ff9-4f37-a4c6-87a1cc63f455" (UID: "9321483f-6ff9-4f37-a4c6-87a1cc63f455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:55:09.448456 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.448438 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:55:09.450165 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.450146 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb" pod "9321483f-6ff9-4f37-a4c6-87a1cc63f455" (UID: "9321483f-6ff9-4f37-a4c6-87a1cc63f455"). InnerVolumeSpecName "kube-api-access-n95sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:55:09.450240 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.450187 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca" pod "9321483f-6ff9-4f37-a4c6-87a1cc63f455" (UID: "9321483f-6ff9-4f37-a4c6-87a1cc63f455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:55:09.549822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.549764 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9321483f-6ff9-4f37-a4c6-87a1cc63f455-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:55:09.549822 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:09.549789 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95sb\" (UniqueName: \"kubernetes.io/projected/9321483f-6ff9-4f37-a4c6-87a1cc63f455-kube-api-access-n95sb\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:55:10.294624 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:10.294585 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" event={"ID":"9321483f-6ff9-4f37-a4c6-87a1cc63f455","Type":"ContainerDied","Data":"36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4"} Nov 02 22:55:10.294777 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:10.294640 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b32c8c977f7a893e81398ddb4d2cba4085ab76efdf8eec0fdb9a0022ff57a4" Nov 02 22:55:10.294777 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:10.294603 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt" Nov 02 22:55:22.497211 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.497151 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:55:22.497988 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.497900 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:55:22.512123 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.512076 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:55:22.512458 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.512407 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:55:22.936276 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.936215 5859 scope.go:122] "RemoveContainer" containerID="7b7b6426c9dda7ec15a9d0f28704349b7f60ceeaf3e81e98eb90f4ad81b8379a" Nov 02 22:55:22.980194 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:22.980164 5859 scope.go:122] "RemoveContainer" containerID="3f60650f0c31b6575f2c2de1682bb7aa80fa4cc3cfca72d0d04a0b8365e709b7" Nov 02 22:55:24.042436 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:24.042391 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns"] Nov 02 22:55:24.088903 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:24.088865 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-n9gns"] Nov 02 22:55:24.570257 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:24.570223 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07029a1f-f4a5-4f57-96a7-0574376c9822" path="/var/lib/kubelet/pods/07029a1f-f4a5-4f57-96a7-0574376c9822/volumes" Nov 02 22:55:25.944700 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:55:25.944664 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:55:41.041809 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:41.041771 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw"] Nov 02 22:55:41.088158 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:41.088123 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-x6tqw"] Nov 02 22:55:42.569709 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:55:42.569673 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160693ee-4f72-4fd9-b729-6c01a878c15e" path="/var/lib/kubelet/pods/160693ee-4f72-4fd9-b729-6c01a878c15e/volumes" Nov 02 22:55:55.944233 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:55:55.944199 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:56:23.108609 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:23.108581 5859 scope.go:122] "RemoveContainer" containerID="061dc136660409f373fdd98ea6daa87aa3e081e1a7da5991f0fbeba16b4bc82c" Nov 02 22:56:23.117268 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:23.117249 5859 scope.go:122] "RemoveContainer" containerID="2079451c37c3a8595fc2acfc667b76e088504b8e5b91d15a6c636e080518ccce" Nov 02 22:56:23.125172 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:23.125155 5859 scope.go:122] "RemoveContainer" containerID="75b6d17fe595828854308b30c3c861ee633070f7d75e91336a295bb252452533" Nov 02 22:56:25.942970 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:56:25.942912 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:56:52.501116 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:52.501076 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:56:52.501755 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:52.501724 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:56:54.554190 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.554156 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q"] Nov 02 22:56:54.554655 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.554636 5859 cpu_manager.go:414] "containerMap: removing container" podUID="9321483f-6ff9-4f37-a4c6-87a1cc63f455" containerName="revoke-keys" Nov 02 22:56:54.554702 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.554658 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9321483f-6ff9-4f37-a4c6-87a1cc63f455" containerName="revoke-keys" Nov 02 22:56:54.554792 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.554782 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="9321483f-6ff9-4f37-a4c6-87a1cc63f455" containerName="revoke-keys" Nov 02 22:56:54.558161 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.558143 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.631701 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.631675 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.631837 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.631723 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.631837 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.631823 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbpc\" (UniqueName: \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.655106 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.655083 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:56:54.655244 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.655224 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:56:54.655299 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.655249 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:56:54.655338 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.655314 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:56:54.657080 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.657062 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q"] Nov 02 22:56:54.733170 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.733143 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbpc\" (UniqueName: \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.733289 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.733260 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.733355 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.733318 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.733892 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.733870 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.735599 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.735579 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.786063 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.786041 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbpc\" (UniqueName: \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\") pod \"maas-api-revoke-sub-d576e68b-qsl6q\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:54.872049 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:54.872013 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:55.239341 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:55.239269 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q"] Nov 02 22:56:55.244726 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:56:55.244702 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7598d6a_e74b_4a1b_a265_5b7a71b78a80.slice/crio-2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950 WatchSource:0}: Error finding container 2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950: Status 404 returned error can't find the container with id 2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950 Nov 02 22:56:55.943366 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:56:55.943333 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:56:55.957486 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:55.957453 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" event={"ID":"e7598d6a-e74b-4a1b-a265-5b7a71b78a80","Type":"ContainerStarted","Data":"5ea5be610eb823865b3df8fd89e8bac0f3f4d9aa996d098716cf8a8601c93a2a"} Nov 02 22:56:55.957621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:55.957509 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" event={"ID":"e7598d6a-e74b-4a1b-a265-5b7a71b78a80","Type":"ContainerStarted","Data":"2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950"} Nov 02 22:56:56.099735 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:56.099691 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" podStartSLOduration=2.099677465 podStartE2EDuration="2.099677465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:56.099206563 +0000 UTC m=+6455.647456343" watchObservedRunningTime="2026-11-02 22:56:56.099677465 +0000 UTC m=+6455.647927305" Nov 02 22:56:56.964734 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:56.964703 5859 generic.go:356] "Generic (PLEG): container finished" podID="e7598d6a-e74b-4a1b-a265-5b7a71b78a80" containerID="5ea5be610eb823865b3df8fd89e8bac0f3f4d9aa996d098716cf8a8601c93a2a" exitCode=0 Nov 02 22:56:56.965128 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:56.964795 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" event={"ID":"e7598d6a-e74b-4a1b-a265-5b7a71b78a80","Type":"ContainerDied","Data":"5ea5be610eb823865b3df8fd89e8bac0f3f4d9aa996d098716cf8a8601c93a2a"} Nov 02 22:56:58.100304 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.100281 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:58.263533 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.263471 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\") pod \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " Nov 02 22:56:58.263674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.263612 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\") pod \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " Nov 02 22:56:58.263674 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.263670 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\" (UniqueName: \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\") pod \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\" (UID: \"e7598d6a-e74b-4a1b-a265-5b7a71b78a80\") " Nov 02 22:56:58.263977 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.263940 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca" pod "e7598d6a-e74b-4a1b-a265-5b7a71b78a80" (UID: "e7598d6a-e74b-4a1b-a265-5b7a71b78a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:58.265475 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.265447 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca" pod "e7598d6a-e74b-4a1b-a265-5b7a71b78a80" (UID: "e7598d6a-e74b-4a1b-a265-5b7a71b78a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:58.265617 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.265598 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc" pod "e7598d6a-e74b-4a1b-a265-5b7a71b78a80" (UID: "e7598d6a-e74b-4a1b-a265-5b7a71b78a80"). InnerVolumeSpecName "kube-api-access-gnbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:58.365284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.365263 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:58.365284 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.365284 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbpc\" (UniqueName: \"kubernetes.io/projected/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-kube-api-access-gnbpc\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:58.365424 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.365295 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7598d6a-e74b-4a1b-a265-5b7a71b78a80-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:58.977373 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.977296 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" Nov 02 22:56:58.977373 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.977333 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q" event={"ID":"e7598d6a-e74b-4a1b-a265-5b7a71b78a80","Type":"ContainerDied","Data":"2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950"} Nov 02 22:56:58.977557 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:56:58.977383 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f5360fe0b5b674ad5cc268ff4cc27e9c3bfab52a443c74ef5dabd8d8e8a3950" Nov 02 22:57:04.042806 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:04.042773 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b"] Nov 02 22:57:04.088978 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:04.088943 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-xrx8b"] Nov 02 22:57:04.569682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:04.569649 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec634eb-f817-4755-8a00-9b2def88bbd0" path="/var/lib/kubelet/pods/9ec634eb-f817-4755-8a00-9b2def88bbd0/volumes" Nov 02 22:57:14.049801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:14.049766 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p"] Nov 02 22:57:14.111972 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:14.092208 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pjh2p"] Nov 02 22:57:14.578785 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:14.578748 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce903ad-a6f4-4231-a479-fd4761121c12" path="/var/lib/kubelet/pods/cce903ad-a6f4-4231-a479-fd4761121c12/volumes" Nov 02 22:57:15.865887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.865860 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87"] Nov 02 22:57:15.866295 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.866282 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e7598d6a-e74b-4a1b-a265-5b7a71b78a80" containerName="revoke-keys" Nov 02 22:57:15.866340 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.866295 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7598d6a-e74b-4a1b-a265-5b7a71b78a80" containerName="revoke-keys" Nov 02 22:57:15.866391 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.866382 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7598d6a-e74b-4a1b-a265-5b7a71b78a80" containerName="revoke-keys" Nov 02 22:57:15.869407 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.869390 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:15.905429 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.905410 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:15.905575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.905439 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:15.905575 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.905469 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgxf\" (UniqueName: \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:15.967567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.967537 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:57:15.967567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.967550 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:57:15.967567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.967560 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jds86\"" Nov 02 22:57:15.967567 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.967571 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:57:15.969597 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:15.969580 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87"] Nov 02 22:57:16.006092 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.006064 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.006218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.006098 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.006218 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.006131 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgxf\" (UniqueName: \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.006679 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.006659 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.008398 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.008376 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.058865 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.058828 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgxf\" (UniqueName: \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\") pod \"maas-api-revoke-sub-d2bc93bc-8fk87\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.181898 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.181848 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:16.548497 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:16.548431 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87"] Nov 02 22:57:16.553779 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:57:16.553750 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12eac97f_7c39_44de_b718_fc58c9a49797.slice/crio-6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193 WatchSource:0}: Error finding container 6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193: Status 404 returned error can't find the container with id 6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193 Nov 02 22:57:17.120069 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:17.120030 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" event={"ID":"12eac97f-7c39-44de-b718-fc58c9a49797","Type":"ContainerStarted","Data":"c95ffae04df450ab5751f7fedb32e034d235beba5fd2425b34d721a884a831fe"} Nov 02 22:57:17.120432 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:17.120089 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" event={"ID":"12eac97f-7c39-44de-b718-fc58c9a49797","Type":"ContainerStarted","Data":"6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193"} Nov 02 22:57:17.228589 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:17.228547 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" podStartSLOduration=2.228533827 podStartE2EDuration="2.228533827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:17.227226768 +0000 UTC m=+6476.775476548" watchObservedRunningTime="2026-11-02 22:57:17.228533827 +0000 UTC m=+6476.776783607" Nov 02 22:57:18.126505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:18.126466 5859 generic.go:356] "Generic (PLEG): container finished" podID="12eac97f-7c39-44de-b718-fc58c9a49797" containerID="c95ffae04df450ab5751f7fedb32e034d235beba5fd2425b34d721a884a831fe" exitCode=0 Nov 02 22:57:18.126897 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:18.126510 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" event={"ID":"12eac97f-7c39-44de-b718-fc58c9a49797","Type":"ContainerDied","Data":"c95ffae04df450ab5751f7fedb32e034d235beba5fd2425b34d721a884a831fe"} Nov 02 22:57:19.266733 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.266708 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:19.335593 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.335569 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\") pod \"12eac97f-7c39-44de-b718-fc58c9a49797\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " Nov 02 22:57:19.335722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.335606 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\") pod \"12eac97f-7c39-44de-b718-fc58c9a49797\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " Nov 02 22:57:19.335722 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.335665 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\" (UniqueName: \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\") pod \"12eac97f-7c39-44de-b718-fc58c9a49797\" (UID: \"12eac97f-7c39-44de-b718-fc58c9a49797\") " Nov 02 22:57:19.335923 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.335900 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca" pod "12eac97f-7c39-44de-b718-fc58c9a49797" (UID: "12eac97f-7c39-44de-b718-fc58c9a49797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:19.336119 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.336096 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-openshift-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:19.337730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.337707 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca" pod "12eac97f-7c39-44de-b718-fc58c9a49797" (UID: "12eac97f-7c39-44de-b718-fc58c9a49797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:19.337730 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.337713 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf" pod "12eac97f-7c39-44de-b718-fc58c9a49797" (UID: "12eac97f-7c39-44de-b718-fc58c9a49797"). InnerVolumeSpecName "kube-api-access-rhgxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:19.437420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.437357 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12eac97f-7c39-44de-b718-fc58c9a49797-maas-api-xks-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:19.437420 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:19.437382 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgxf\" (UniqueName: \"kubernetes.io/projected/12eac97f-7c39-44de-b718-fc58c9a49797-kube-api-access-rhgxf\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:20.140366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:20.140333 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" event={"ID":"12eac97f-7c39-44de-b718-fc58c9a49797","Type":"ContainerDied","Data":"6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193"} Nov 02 22:57:20.140538 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:20.140372 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87" Nov 02 22:57:20.140538 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:20.140389 5859 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6975926b18c4fcd422e04eec5d2c9c9eee1454a9062baf9b5c82940ef4077193" Nov 02 22:57:23.155881 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:23.155846 5859 scope.go:122] "RemoveContainer" containerID="cc0ada6b6fec5fbfa26809ca689155de1401fad136edfd5c610cb8240537448c" Nov 02 22:57:23.164366 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:23.164341 5859 scope.go:122] "RemoveContainer" containerID="6cf983163e6867feca3abe526887733b361f5211ca15631e2c977cb27a182c59" Nov 02 22:57:23.172028 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:23.172009 5859 scope.go:122] "RemoveContainer" containerID="6c13ad1e1711b22e54c7c3f44f905037755ba8b3b5674a6dd16b7b792cce875e" Nov 02 22:57:25.943572 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:57:25.943536 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:57:28.813569 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.813533 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:28.814339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.814316 5859 cpu_manager.go:414] "containerMap: removing container" podUID="12eac97f-7c39-44de-b718-fc58c9a49797" containerName="revoke-keys" Nov 02 22:57:28.814406 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.814343 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12eac97f-7c39-44de-b718-fc58c9a49797" containerName="revoke-keys" Nov 02 22:57:28.814468 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.814457 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="12eac97f-7c39-44de-b718-fc58c9a49797" containerName="revoke-keys" Nov 02 22:57:28.821868 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.821832 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:28.826527 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.826503 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6"] Nov 02 22:57:28.830997 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.830974 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:28.875907 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.875877 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 22:57:28.876043 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.875932 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:57:28.925068 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925039 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-metrics-certs\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:28.925214 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925096 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:28.925266 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925198 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsnv\" (UniqueName: \"kubernetes.io/projected/765ffffc-fecf-4549-a636-1c25d5680174-kube-api-access-mxsnv\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:28.925339 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925323 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fn4r\" (UniqueName: \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:28.925403 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925378 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-maas-api-tls\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:28.925452 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925442 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765ffffc-fecf-4549-a636-1c25d5680174-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:28.925509 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925494 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:28.925571 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925559 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:28.925731 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925707 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:57:28.925840 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925736 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:57:28.925840 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.925769 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:57:28.927905 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.927881 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:28.929910 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:28.929885 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6"] Nov 02 22:57:29.026187 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026158 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.026301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026200 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-metrics-certs\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.026301 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026228 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.026423 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026406 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsnv\" (UniqueName: \"kubernetes.io/projected/765ffffc-fecf-4549-a636-1c25d5680174-kube-api-access-mxsnv\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.026502 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026484 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fn4r\" (UniqueName: \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.026562 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026542 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-maas-api-tls\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.026627 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026582 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765ffffc-fecf-4549-a636-1c25d5680174-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.026682 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.026622 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.027315 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.027295 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.027447 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.027427 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/765ffffc-fecf-4549-a636-1c25d5680174-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.028622 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.028595 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.028789 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.028771 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.028887 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.028872 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-metrics-certs\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.029002 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.028984 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/765ffffc-fecf-4549-a636-1c25d5680174-maas-api-tls\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.083510 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.083490 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsnv\" (UniqueName: \"kubernetes.io/projected/765ffffc-fecf-4549-a636-1c25d5680174-kube-api-access-mxsnv\") pod \"maas-api-7c9bd99f4f-ltnm6\" (UID: \"765ffffc-fecf-4549-a636-1c25d5680174\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.083607 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.083593 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fn4r\" (UniqueName: \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\") pod \"maas-api-bbf944969-wt8rf\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.138048 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.138024 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:29.146055 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.146032 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:29.515408 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.515384 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:29.517800 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:57:29.517773 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69a0040_235c_409f_9077_dd01e24a5f40.slice/crio-fa30ab7f3d0335936ba62578f0e57b1e9a04f0a65dc45117c1dec9adeda4ab2e WatchSource:0}: Error finding container fa30ab7f3d0335936ba62578f0e57b1e9a04f0a65dc45117c1dec9adeda4ab2e: Status 404 returned error can't find the container with id fa30ab7f3d0335936ba62578f0e57b1e9a04f0a65dc45117c1dec9adeda4ab2e Nov 02 22:57:29.560796 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:29.560773 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6"] Nov 02 22:57:29.562297 ip-10-0-137-1 kubenswrapper[5859]: W1102 22:57:29.562276 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765ffffc_fecf_4549_a636_1c25d5680174.slice/crio-b0277155f774fa9a06612bd778dfe28307d33fa7f69c7e1e2dbe7751850eafe8 WatchSource:0}: Error finding container b0277155f774fa9a06612bd778dfe28307d33fa7f69c7e1e2dbe7751850eafe8: Status 404 returned error can't find the container with id b0277155f774fa9a06612bd778dfe28307d33fa7f69c7e1e2dbe7751850eafe8 Nov 02 22:57:30.196716 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:30.196687 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" event={"ID":"e69a0040-235c-409f-9077-dd01e24a5f40","Type":"ContainerStarted","Data":"fa30ab7f3d0335936ba62578f0e57b1e9a04f0a65dc45117c1dec9adeda4ab2e"} Nov 02 22:57:30.198028 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:30.197978 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" event={"ID":"765ffffc-fecf-4549-a636-1c25d5680174","Type":"ContainerStarted","Data":"b0277155f774fa9a06612bd778dfe28307d33fa7f69c7e1e2dbe7751850eafe8"} Nov 02 22:57:31.205333 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.205294 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" event={"ID":"e69a0040-235c-409f-9077-dd01e24a5f40","Type":"ContainerStarted","Data":"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4"} Nov 02 22:57:31.205801 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.205378 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:31.206974 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.206936 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" event={"ID":"765ffffc-fecf-4549-a636-1c25d5680174","Type":"ContainerStarted","Data":"501eae005ef8411576bfebb9ee92cc7f28f61cc831e80e1e7d4773d7d0b7a5b9"} Nov 02 22:57:31.207112 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.206995 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:31.320155 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.320104 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" podStartSLOduration=2.325897807 podStartE2EDuration="3.320090524s" totalImagesPullingTime="994.192717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:31.318561163 +0000 UTC m=+6490.866810962" watchObservedRunningTime="2026-11-02 22:57:31.320090524 +0000 UTC m=+6490.868340686" Nov 02 22:57:31.432564 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:31.432511 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" podStartSLOduration=2.46116853 podStartE2EDuration="3.432499655s" totalImagesPullingTime="971.331125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:31.432102909 +0000 UTC m=+6490.980352689" watchObservedRunningTime="2026-11-02 22:57:31.432499655 +0000 UTC m=+6490.980749435" Nov 02 22:57:37.217871 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.217801 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-ltnm6" Nov 02 22:57:37.218313 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.218218 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:37.384430 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.384400 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:37.384634 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.384569 5859 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" podUID="e69a0040-235c-409f-9077-dd01e24a5f40" containerName="maas-api" containerID="cri-o://5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4" gracePeriod=30 Nov 02 22:57:37.635457 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.635427 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:37.707413 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.707390 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\" (UniqueName: \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\") pod \"e69a0040-235c-409f-9077-dd01e24a5f40\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " Nov 02 22:57:37.707536 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.707512 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\") pod \"e69a0040-235c-409f-9077-dd01e24a5f40\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " Nov 02 22:57:37.707582 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.707560 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\") pod \"e69a0040-235c-409f-9077-dd01e24a5f40\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " Nov 02 22:57:37.707621 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.707602 5859 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\") pod \"e69a0040-235c-409f-9077-dd01e24a5f40\" (UID: \"e69a0040-235c-409f-9077-dd01e24a5f40\") " Nov 02 22:57:37.708035 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.708007 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca" pod "e69a0040-235c-409f-9077-dd01e24a5f40" (UID: "e69a0040-235c-409f-9077-dd01e24a5f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:37.709505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.709482 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls" pod "e69a0040-235c-409f-9077-dd01e24a5f40" (UID: "e69a0040-235c-409f-9077-dd01e24a5f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:37.709505 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.709499 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r" pod "e69a0040-235c-409f-9077-dd01e24a5f40" (UID: "e69a0040-235c-409f-9077-dd01e24a5f40"). InnerVolumeSpecName "kube-api-access-5fn4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:37.709635 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.709545 5859 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs" pod "e69a0040-235c-409f-9077-dd01e24a5f40" (UID: "e69a0040-235c-409f-9077-dd01e24a5f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:37.808480 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.808420 5859 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-metrics-certs\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:37.808480 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.808445 5859 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e69a0040-235c-409f-9077-dd01e24a5f40-maas-api-tls\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:37.808480 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.808455 5859 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69a0040-235c-409f-9077-dd01e24a5f40-openshift-service-ca\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:37.808480 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:37.808463 5859 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fn4r\" (UniqueName: \"kubernetes.io/projected/e69a0040-235c-409f-9077-dd01e24a5f40-kube-api-access-5fn4r\") on node \"ip-10-0-137-1.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:38.245760 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.245727 5859 generic.go:356] "Generic (PLEG): container finished" podID="e69a0040-235c-409f-9077-dd01e24a5f40" containerID="5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4" exitCode=0 Nov 02 22:57:38.246156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.245774 5859 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" Nov 02 22:57:38.246156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.245809 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" event={"ID":"e69a0040-235c-409f-9077-dd01e24a5f40","Type":"ContainerDied","Data":"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4"} Nov 02 22:57:38.246156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.245863 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-wt8rf" event={"ID":"e69a0040-235c-409f-9077-dd01e24a5f40","Type":"ContainerDied","Data":"fa30ab7f3d0335936ba62578f0e57b1e9a04f0a65dc45117c1dec9adeda4ab2e"} Nov 02 22:57:38.246156 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.245884 5859 scope.go:122] "RemoveContainer" containerID="5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4" Nov 02 22:57:38.255309 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.255293 5859 scope.go:122] "RemoveContainer" containerID="5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4" Nov 02 22:57:38.257398 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:57:38.257379 5859 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4\": container with ID starting with 5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4 not found: ID does not exist" containerID="5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4" Nov 02 22:57:38.257471 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.257407 5859 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4"} err="failed to get container status \"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4\": rpc error: code = NotFound desc = could not find container \"5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4\": container with ID starting with 5c8c0440e136a3d6d07fd210d8020441663491634a90feb7272bf0db7cf9f2f4 not found: ID does not exist" Nov 02 22:57:38.410445 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.410419 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:38.415975 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.415939 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-wt8rf"] Nov 02 22:57:38.570144 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:57:38.570116 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69a0040-235c-409f-9077-dd01e24a5f40" path="/var/lib/kubelet/pods/e69a0040-235c-409f-9077-dd01e24a5f40/volumes" Nov 02 22:57:55.945568 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:57:55.945529 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:58:23.199699 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:58:23.199666 5859 scope.go:122] "RemoveContainer" containerID="ba7fa966511c11c0674af0fda5e76d93e10c379130862b073d4adccf1648d92e" Nov 02 22:58:23.208783 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:58:23.208766 5859 scope.go:122] "RemoveContainer" containerID="2d2335f6f77da282b7b12451ba5dedb2e4668cd25835c3eb83d446dedef53663" Nov 02 22:58:26.033356 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:58:26.033300 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:58:55.942544 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:58:55.942503 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:59:23.234991 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:59:23.234947 5859 scope.go:122] "RemoveContainer" containerID="9393db7b0ed89cafbc001cd1b3624ac3d09cb527501c60289af72321f27a090e" Nov 02 22:59:25.942986 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:59:25.942930 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 22:59:52.493077 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:59:52.493035 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:59:52.493809 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:59:52.493774 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:59:52.503195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:59:52.503082 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:59:52.503195 ip-10-0-137-1 kubenswrapper[5859]: I1102 22:59:52.503136 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:59:55.944535 ip-10-0-137-1 kubenswrapper[5859]: E1102 22:59:55.944488 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:00:10.048709 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:00:10.048669 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt"] Nov 02 23:00:10.098467 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:00:10.098433 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t6tdt"] Nov 02 23:00:10.722976 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:00:10.719401 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9321483f-6ff9-4f37-a4c6-87a1cc63f455" path="/var/lib/kubelet/pods/9321483f-6ff9-4f37-a4c6-87a1cc63f455/volumes" Nov 02 23:00:25.943716 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:00:25.943677 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:00:55.946035 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:00:55.945912 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:01:07.494147 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:07.494089 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:01:07.496561 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:07.494912 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:01:07.500190 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:07.500050 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:01:07.500190 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:07.500103 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:01:23.265351 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:23.265318 5859 scope.go:122] "RemoveContainer" containerID="b8df5e693cf4bd95b05ec21d8e5c39cf768fbeb9f88db1c975640686c63b6846" Nov 02 23:01:25.943722 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:01:25.943686 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:01:55.319496 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:55.319031 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:01:55.319496 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:55.319106 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:01:56.010836 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:01:56.010789 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:01:59.045122 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:59.045085 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q"] Nov 02 23:01:59.095365 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:01:59.095329 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qsl6q"] Nov 02 23:02:00.571078 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:00.571047 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7598d6a-e74b-4a1b-a265-5b7a71b78a80" path="/var/lib/kubelet/pods/e7598d6a-e74b-4a1b-a265-5b7a71b78a80/volumes" Nov 02 23:02:20.048062 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:20.048026 5859 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87"] Nov 02 23:02:20.098506 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:20.098478 5859 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8fk87"] Nov 02 23:02:20.569907 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:20.569877 5859 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12eac97f-7c39-44de-b718-fc58c9a49797" path="/var/lib/kubelet/pods/12eac97f-7c39-44de-b718-fc58c9a49797/volumes" Nov 02 23:02:25.945053 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:02:25.945017 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:02:37.504104 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:37.504062 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-j9dwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:02:37.504811 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:37.504135 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7zm8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:02:37.504991 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:37.504861 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-j9dwx" podUID="0914efbd-e8c2-4055-922b-760acb575c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:02:37.505207 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:02:37.505176 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7zm8m" podUID="a3fff690-40c3-413e-98d5-a4955dda45b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:02:55.943232 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:02:55.943198 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:03:10.320102 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:10.320045 5859 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-5chzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:03:10.320904 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:10.320852 5859 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-5chzz" podUID="befb4e51-1fca-4d2b-9247-4eebca3cd2f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:03:23.295442 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:23.295411 5859 scope.go:122] "RemoveContainer" containerID="c95ffae04df450ab5751f7fedb32e034d235beba5fd2425b34d721a884a831fe" Nov 02 23:03:23.304082 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:23.304064 5859 scope.go:122] "RemoveContainer" containerID="5ea5be610eb823865b3df8fd89e8bac0f3f4d9aa996d098716cf8a8601c93a2a" Nov 02 23:03:25.942518 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:03:25.942483 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:03:55.944078 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:03:55.944043 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:03:56.033543 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:56.033508 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-t7cvb_3632f233-0d30-47af-afb3-0d59fd3444c9/manager/0.log" Nov 02 23:03:56.549545 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:56.549516 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mqd69_5a272ed8-590a-4bc5-aace-1f092f17eb9d/manager/0.log" Nov 02 23:03:58.100154 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:03:58.100124 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c9bd99f4f-ltnm6_765ffffc-fecf-4549-a636-1c25d5680174/maas-api/0.log" Nov 02 23:04:04.104823 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:04.104791 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vsgm4_33a98961-7c4a-4fdd-a2d9-b1072164286d/manager/0.log" Nov 02 23:04:04.314181 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:04.314153 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qr5f4_28db1cfd-ba37-4772-91fb-ad9f035e3624/kuadrant-console-plugin/0.log" Nov 02 23:04:06.752833 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:06.752800 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78fbf7b575-mt2tb_e594ed53-c428-4eb4-9410-cca9c7a23763/kube-auth-proxy/0.log" Nov 02 23:04:07.461584 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:07.461555 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-5chzz_befb4e51-1fca-4d2b-9247-4eebca3cd2f3/istio-proxy/0.log" Nov 02 23:04:07.970838 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:07.970813 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zm8m_a3fff690-40c3-413e-98d5-a4955dda45b3/istio-proxy/0.log" Nov 02 23:04:08.710103 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:08.710073 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-j9dwx_0914efbd-e8c2-4055-922b-760acb575c85/istio-proxy/0.log" Nov 02 23:04:09.401063 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:09.401033 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-tt978_e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2/istio-proxy/0.log" Nov 02 23:04:11.601830 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:11.601803 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4kjmm_1113b3ed-88cb-4927-9acd-182acfc8c366/main/0.log" Nov 02 23:04:11.845160 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:11.845137 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bvq5s_a6280fe2-f94e-449b-9eed-18159ae4611f/main/0.log" Nov 02 23:04:12.467278 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:12.467251 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8svm7x_480e96b6-2aae-45f4-b3d3-98840fb956f6/main/0.log" Nov 02 23:04:13.066068 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:13.066043 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-zwn7n_8e0ac144-950e-424b-9461-6a2333d05969/main/0.log" Nov 02 23:04:25.942818 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:04:25.942783 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:04:32.100530 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:32.100498 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xm28p_fddf083e-247f-4cdc-9722-3e6c2b43dd50/global-pull-secret-syncer/0.log" Nov 02 23:04:35.490116 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:35.490089 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbbpt_04d04d90-6021-4329-b47d-a41847969eef/konnectivity-agent/0.log" Nov 02 23:04:35.732875 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:35.732847 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-t7cvb_3632f233-0d30-47af-afb3-0d59fd3444c9/manager/0.log" Nov 02 23:04:36.678854 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:36.678817 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-1.us-west-2.compute.internal_aedbdf8e991546e8774cd514478921e1/haproxy/0.log" Nov 02 23:04:36.811471 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:36.811445 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mqd69_5a272ed8-590a-4bc5-aace-1f092f17eb9d/manager/0.log" Nov 02 23:04:41.536761 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.536725 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-t7cvb_3632f233-0d30-47af-afb3-0d59fd3444c9/manager/0.log" Nov 02 23:04:41.643550 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.643516 5859 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj"] Nov 02 23:04:41.643916 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.643903 5859 cpu_manager.go:414] "containerMap: removing container" podUID="e69a0040-235c-409f-9077-dd01e24a5f40" containerName="maas-api" Nov 02 23:04:41.643916 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.643916 5859 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69a0040-235c-409f-9077-dd01e24a5f40" containerName="maas-api" Nov 02 23:04:41.644074 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.644027 5859 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69a0040-235c-409f-9077-dd01e24a5f40" containerName="maas-api" Nov 02 23:04:41.647467 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.647447 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.697195 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.697168 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-proc\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.697316 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.697220 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-sys\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.697316 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.697236 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttkr\" (UniqueName: \"kubernetes.io/projected/c8ba0a99-a83c-4607-8914-543de130b47a-kube-api-access-4ttkr\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.697453 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.697377 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-lib-modules\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.697553 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.697532 5859 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-podres\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.699484 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.699462 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"openshift-service-ca.crt\"" Nov 02 23:04:41.699603 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.699555 5859 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"kube-root-ca.crt\"" Nov 02 23:04:41.750110 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.750083 5859 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlpkd\"/\"default-dockercfg-v24v2\"" Nov 02 23:04:41.752192 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.752154 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj"] Nov 02 23:04:41.798606 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798550 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttkr\" (UniqueName: \"kubernetes.io/projected/c8ba0a99-a83c-4607-8914-543de130b47a-kube-api-access-4ttkr\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798606 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798589 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-lib-modules\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798736 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798639 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-podres\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798736 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798679 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-proc\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798816 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798730 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-lib-modules\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798816 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798745 5859 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-sys\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798816 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798785 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-podres\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798816 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798802 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-proc\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.798990 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.798819 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba0a99-a83c-4607-8914-543de130b47a-sys\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.855356 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.855334 5859 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttkr\" (UniqueName: \"kubernetes.io/projected/c8ba0a99-a83c-4607-8914-543de130b47a-kube-api-access-4ttkr\") pod \"perf-node-gather-daemonset-vvlpj\" (UID: \"c8ba0a99-a83c-4607-8914-543de130b47a\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:41.960856 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:41.960832 5859 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:42.339601 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:42.339574 5859 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj"] Nov 02 23:04:42.343975 ip-10-0-137-1 kubenswrapper[5859]: W1102 23:04:42.343918 5859 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8ba0a99_a83c_4607_8914_543de130b47a.slice/crio-97f4a888aa0ca27d23ba55a40affd3e83c0f8a66b19df4451d854adf4b96118b WatchSource:0}: Error finding container 97f4a888aa0ca27d23ba55a40affd3e83c0f8a66b19df4451d854adf4b96118b: Status 404 returned error can't find the container with id 97f4a888aa0ca27d23ba55a40affd3e83c0f8a66b19df4451d854adf4b96118b Nov 02 23:04:42.394639 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:42.394605 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" event={"ID":"c8ba0a99-a83c-4607-8914-543de130b47a","Type":"ContainerStarted","Data":"97f4a888aa0ca27d23ba55a40affd3e83c0f8a66b19df4451d854adf4b96118b"} Nov 02 23:04:42.667268 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:42.667198 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mqd69_5a272ed8-590a-4bc5-aace-1f092f17eb9d/manager/0.log" Nov 02 23:04:42.765551 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:42.765525 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vsgm4_33a98961-7c4a-4fdd-a2d9-b1072164286d/manager/0.log" Nov 02 23:04:43.170285 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:43.170262 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qr5f4_28db1cfd-ba37-4772-91fb-ad9f035e3624/kuadrant-console-plugin/0.log" Nov 02 23:04:43.400985 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:43.400937 5859 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" event={"ID":"c8ba0a99-a83c-4607-8914-543de130b47a","Type":"ContainerStarted","Data":"0c30cb4d286e276a8f9646752bef0ddb28c10ec4dd6f680b6bb88cfbda2135d8"} Nov 02 23:04:43.401144 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:43.401022 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:43.513781 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:43.513706 5859 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" podStartSLOduration=2.513695613 podStartE2EDuration="2.513695613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:43.512654138 +0000 UTC m=+6923.060903922" watchObservedRunningTime="2026-11-02 23:04:43.513695613 +0000 UTC m=+6923.061945394" Nov 02 23:04:47.067128 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:47.067099 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/alertmanager/0.log" Nov 02 23:04:47.069416 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:47.069381 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-t7cvb_3632f233-0d30-47af-afb3-0d59fd3444c9/manager/0.log" Nov 02 23:04:47.516634 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:47.516566 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/config-reloader/0.log" Nov 02 23:04:47.920509 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:47.920480 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/kube-rbac-proxy-web/0.log" Nov 02 23:04:48.164057 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:48.164026 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mqd69_5a272ed8-590a-4bc5-aace-1f092f17eb9d/manager/0.log" Nov 02 23:04:48.324531 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:48.324508 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/kube-rbac-proxy/0.log" Nov 02 23:04:48.726762 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:48.726677 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/kube-rbac-proxy-metric/0.log" Nov 02 23:04:49.131643 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:49.131623 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/prom-label-proxy/0.log" Nov 02 23:04:49.416276 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:49.416217 5859 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-vvlpj" Nov 02 23:04:49.534964 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:49.534930 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbe35f1-c94d-4036-b80f-449bf85d5b7a/init-config-reloader/0.log" Nov 02 23:04:50.431451 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:50.431422 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cs4v2_c812be04-0eac-4a4c-9456-60332f6fa93d/kube-state-metrics/0.log" Nov 02 23:04:50.830116 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:50.830088 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cs4v2_c812be04-0eac-4a4c-9456-60332f6fa93d/kube-rbac-proxy-main/0.log" Nov 02 23:04:51.234224 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:51.234157 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cs4v2_c812be04-0eac-4a4c-9456-60332f6fa93d/kube-rbac-proxy-self/0.log" Nov 02 23:04:51.641851 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:51.641829 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b797d4856-8xgjr_1a9bc7cc-f7b4-45c8-a099-ba0d03a1b2c1/metrics-server/0.log" Nov 02 23:04:52.045661 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:52.045594 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-99xw5_0423cd6d-c791-4383-92cd-8b699aca0c0e/monitoring-plugin/0.log" Nov 02 23:04:52.455867 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:52.455842 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6phbr_93eb21de-12ad-49fb-b162-b53f3763d2ce/node-exporter/0.log" Nov 02 23:04:52.860014 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:52.859988 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6phbr_93eb21de-12ad-49fb-b162-b53f3763d2ce/kube-rbac-proxy/0.log" Nov 02 23:04:53.267180 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:53.267120 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6phbr_93eb21de-12ad-49fb-b162-b53f3763d2ce/init-textfile/0.log" Nov 02 23:04:53.621211 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:53.621188 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78fbf7b575-mt2tb_e594ed53-c428-4eb4-9410-cca9c7a23763/kube-auth-proxy/0.log" Nov 02 23:04:54.246576 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:54.246548 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-5chzz_befb4e51-1fca-4d2b-9247-4eebca3cd2f3/istio-proxy/0.log" Nov 02 23:04:54.672827 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:54.672775 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-7zm8m_a3fff690-40c3-413e-98d5-a4955dda45b3/istio-proxy/0.log" Nov 02 23:04:55.356208 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:55.356177 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-j9dwx_0914efbd-e8c2-4055-922b-760acb575c85/istio-proxy/0.log" Nov 02 23:04:55.945742 ip-10-0-137-1 kubenswrapper[5859]: E1102 23:04:55.945710 5859 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tgsfm" containerName="extractor" Nov 02 23:04:55.989199 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:55.989175 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-tt978_e6ce7041-ca7a-4743-9020-b3f7d7a9fbf2/istio-proxy/0.log" Nov 02 23:04:56.153768 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:56.153742 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2l69_992df832-d24f-4517-ab42-d47ff9f51b7e/kube-rbac-proxy-main/0.log" Nov 02 23:04:56.555519 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:56.555496 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2l69_992df832-d24f-4517-ab42-d47ff9f51b7e/kube-rbac-proxy-self/0.log" Nov 02 23:04:56.956842 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:56.956742 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v2l69_992df832-d24f-4517-ab42-d47ff9f51b7e/openshift-state-metrics/0.log" Nov 02 23:04:57.381282 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:57.381254 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/prometheus/0.log" Nov 02 23:04:57.782981 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:57.782901 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/config-reloader/0.log" Nov 02 23:04:58.185345 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:58.185324 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/thanos-sidecar/0.log" Nov 02 23:04:58.587487 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:58.587465 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/kube-rbac-proxy-web/0.log" Nov 02 23:04:58.990426 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:58.990361 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/kube-rbac-proxy/0.log" Nov 02 23:04:59.393629 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:59.393603 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/kube-rbac-proxy-thanos/0.log" Nov 02 23:04:59.800482 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:04:59.800416 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc420536-56b5-400c-8d7a-fcefa442eaec/init-config-reloader/0.log" Nov 02 23:05:01.433355 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:01.433328 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8589895978-xdksc_53127e36-620e-41bb-940f-e4548cebc2ee/telemeter-client/0.log" Nov 02 23:05:01.836858 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:01.836832 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8589895978-xdksc_53127e36-620e-41bb-940f-e4548cebc2ee/reload/0.log" Nov 02 23:05:02.240978 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:02.240904 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8589895978-xdksc_53127e36-620e-41bb-940f-e4548cebc2ee/kube-rbac-proxy/0.log" Nov 02 23:05:02.527382 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:02.527323 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-vsgm4_33a98961-7c4a-4fdd-a2d9-b1072164286d/manager/0.log" Nov 02 23:05:02.648644 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:02.648622 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/thanos-query/0.log" Nov 02 23:05:02.736493 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:02.736471 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qr5f4_28db1cfd-ba37-4772-91fb-ad9f035e3624/kuadrant-console-plugin/0.log" Nov 02 23:05:03.049316 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:03.049295 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/kube-rbac-proxy-web/0.log" Nov 02 23:05:03.450731 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:03.450702 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/kube-rbac-proxy/0.log" Nov 02 23:05:03.862741 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:03.862715 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/prom-label-proxy/0.log" Nov 02 23:05:04.265105 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:04.265034 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/kube-rbac-proxy-rules/0.log" Nov 02 23:05:04.668367 ip-10-0-137-1 kubenswrapper[5859]: I1102 23:05:04.668334 5859 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d55f57bf6-46r5k_1e77bc5a-0603-4b53-900d-c3335549d101/kube-rbac-proxy-metrics/0.log"